10.1. Progressive Validation Technique

Progressive validation is a technique which allows you to use almost $\frac{1}{2}$ of the data in a holdout set for training purposes while still providing the same guarantee as the holdout bound. It first appeared in [3] and is discussed in a more refined and detailed form here.

Suppose that you have a training set of size $m_{\text{train}}$ and test set of size $m_{\text{pv}}$. Progressive validation starts by first learning a hypothesis on the training set and then testing on the first example of the test set. Then, we train on training set plus the first example of the test set and test on the second example of the test set. The process continues $m_{\text{test}}$ iterations. Let $m$ abbreviate $m_{\text{pv}}$. Then, we have $m$ hypotheses, $h_1,...,h_m$ and $m$ error observations, ${\widehat{e}}_1,...,{\widehat{e}}_m$. The hypothesis output by progressive validation is the randomized hypothesis which chooses uniformly from $h_1,...,h_m$ and evaluates to get an estimated output. Note that this protocol is similar to those in [36] and the new thing here is an analysis of performance.

Since we are randomizing over hypotheses trained on $m_{\text{train}}$ to $m_{\text{train}}+m_{\text{pv}}-1$ examples, the expected number of examples used by any hypothesis is $m_{\text{train}}+\frac{m_{\text{pv}}-1}{2}$. Given that training can exhibit phase transitions, the extra few examples can greatly improve the accuracy of the trained example.

Viewed as an interactive proof of learning, the progressive validation technique follows the protocol of figure 10.1.1.

The true error rate of this randomized hypothesis will be: $$e_{\text{pv}}=\frac{1}{m_{\text{pv}}}{\sum }_{i=1}^{m_{\text{pv}}}e\left(h_i\right)$$ where $e\left(h_i\right)={Pr}_D\left(h_i\left(x\right)\ne y\right)$ and the empirical error estimate of this randomized hypothesis will be: $${\widehat{e}}_{\text{pv}}=\frac{1}{m_{\text{pv}}}{\sum }_{i=1}^{m_{\text{pv}}}{\widehat{e}}_i$$