4.1 Simple Holdout

4.1. Simple Holdout

The simplest bound arises for the classical technique of splitting the data set into two pieces: a training set of size $m_{train}$ and a test set of size $m_{test}$ . In this setting, the following simple bound applies:

THEOREM 4.1.1.

(Holdout Sample Complexity) Let ${\hat{e}}_{test} (h)$ be the empirical error on the test set and $e_{D} (h)$ be the true error rate of the hypothesis, then we have: $\forall h {Pr}_{D^{m}} (e_{D} (h) \geq \bar{e} (m_{test}, {\hat{e}}_{test} (h), δ)) \leq δ$ where $\bar{e} (m, \frac{k}{m}, δ) \equiv {max}_{p} {p : Bin (m, k, p) = δ}$

PROOF. The proof is just a simple identification with the Binomial. For any distribution over $(x, y)$ pairs and any hypothesis, $h$ , there exists some probability, $e_{D} (h)$ , that the hypothesis predicts incorrectly. We can regard this event as a coin flip with bias $e_{D} (h)$ . Since each example is picked independently, the distribution of the empirical error rate will then be a Binomial distribution. Given that the distribution is Binomial we calculate an upper bound which holds with high probability. ▫

There are two immediate corollaries of the holdout theorem ( 4.1.1) which are mathematically simpler although not as tight. The first corollary applies to the limited “realizable” setting where you happen to observe $0$ test errors.

COROLLARY 4.1.2.

(Realizable Holdout Sample Complexity) $\forall h {Pr}_{D} ({\hat{e}}_{test} (h) = 0 ∣ e_{D} (h) \geq \frac{ln \frac{1}{δ}}{m_{test}}) \leq δ$

PROOF. Specializing theorem 4.1.1 to the zero empirical error case, we get: $Bin (m_{test}, 0, ε) = {(1 - ε)}^{m_{test}} \leq e^{- ε m_{test}}$ Setting this equal to $δ$ and solving for $ε$ gives us the result. ▫

A second corollary applies to all results, not just those where we observe $0$ errors.

COROLLARY 4.1.3.

(Agnostic Holdout Sample Complexity) $\forall h {Pr}_{D} (e_{D} (h) - {\hat{e}}_{test} (h) \geq \sqrt{\frac{ln \frac{1}{δ}}{2 m_{test}}}) \leq δ$

PROOF. Loosening theorem 4.1.1 with the Hoeffding approximation for $\frac{k}{m_{test}} < ε$ , we get: $Bin (m_{test}, k, ε) \leq e^{- 2 m_{test} {(ε - \frac{k}{m_{test}})}^{2}}$ Using the inversion lemma 3.4.1 we can set this equal to $δ$ , and solve for $ε$ to get the result. ▫

REMARK 4.1.4.

Similar theorems apply to bound ${\hat{e}}_{test} (h) - e_{D} (h)$ .

How tight is the test sample complexity theorem 4.1.1? The answer is very tight. Let us define $\bar{e} (m, \hat{e} (h), δ) \equiv {max}_{p} p : Bin (m_{test}, m_{test} {\hat{e}}_{S} (h), p) \geq δ$ as our true error bound. We wish to know how much $e_{D} (h)$ and $\bar{e} (m_{test}, \hat{e} (h), δ)$ differ. Applying the Hoeffding approximation, we know that with high probability, $e_{D} (h) \geq \bar{e} (m_{test}, \hat{e} (h), δ) - 2 \sqrt{\frac{ln \frac{1}{δ}}{2 m_{test}}}$ . Thus the region in which $e_{D} (h)$ is confined with high confidence is of size $2 \sqrt{\frac{ln \frac{1}{δ}}{2 m_{test}}}$ or smaller.

It is common practice in the field of machine learning to use the gaussian approximation in reporting error bars. The practice is reasonably safe because it is usually pessimistic. However, this can occasionally lead to embarrassing results where error rates such as $0.01 \pm 0.02$ are reported. The test sample complexity theorem never produces an upper bound greater than $1$ or lower bound less than $0$ because it uses the fundamental Binomial distribution. This approach is the “right” way to report test-set based errors, given the assumption of independence. Appendix Section 16.1 documents how to apply this bound. Pictorially we can represent this as in figure 4.1.1.

Figure 4.1.1:

For this diagram “increasing time” is pointing downwards. The only requirement for applying this bound is that the learner must commit to a hypothesis without knowledge of the test examples. Similar diagrams for other bounds will be presented later (and they are somewhat more complicated). We can think of the bound as a technique by which the “Learner” can convince the “Verifier” that learning has occurred. Each of the proofs in this thesis can be thought of as a communication protocol for an interactive proof of learning by the Learner.

Some results for application of the simple test set bound are presented on page 326 in figure 12.3.3. In summary, the test set bound tends to work quite well (in practice) when sufficient examples are available.

Given that the bounds for the simple holdout technique are so tight, why do we need to engage in further work? There is one serious drawback to the holdout technique—application of the holdout technique requires $m_{test}$ otherwise unused examples. This can strongly degrade the value of the learned hypothesis because an extra $m_{test}$ examples for the training set could reduce the true error of the learned hypothesis from $0.5$ to $0.0$ on some learning problems.

There is another reason why training set based bounds are important. Many learning algorithms implicitly assume that the training error “behaves like” the true error in choosing the hypothesis. With an inadequate number of training examples, there may be very little relationship between the behavior of the training error and the true error. Training error based bounds can be used in the training algorithm.

There are two basic approaches to this difficulty:

Try to reduce $m_{test}$ using more sophisticated holdout techniques.
Do not use a holdout set. Instead, train and test on the same set of examples using a more sophisticated bound.

Before discussing approach (2) we will make a few comments about approach (1) to suggest the variety of theoretical difficulties which occur when using approach (1).

4.1.1. Cross Validation

One of the standard techniques for attempting to improve on the holdout bound is cross validation. K-fold cross validation divides the data into $K$ folds of size $\frac{m}{K}$ (assume $m$ is divisible by $K$ for simplicity). Then, for every fold $i$ , holdout fold $i$ , train on the remainder of the data and test on fold $i$ . Let the hypotheses we found by training be known as $h_{1}, . . ., h_{K}$ and their respective holdout errors as ${\hat{e}}_{1}, . . ., {\hat{e}}_{K}$ . Also let ${\hat{e}}_{c v} = \frac{1}{K} \sum_{i = 1}^{K} {\hat{e}}_{i}$ .

There are several variations of cross validation. If $K = m$ , the procedure is often called “leave one out cross validation”. In one variant, you train on all of the data to learn a new hypothesis, $h$ , and assume a true error rate near ${\hat{e}}_{C V}$ . In another variant, you predict according to $h_{c v} = U n i f o r m (h_{1}, . . ., h_{K})$ . The latter variant is simpler to analyze because linearity of expectation implies that ${\hat{e}}_{c v}$ is an unbiased estimate of ${\bar{e}}_{c v}$ .

There are strong results known for cross validation on nearest-neighbor, kernel, and histogram classifiers [11]. In general, only very weak results are known about bounds on the variance of cross validation for general classifiers. The “general” results include “Sanity check bounds” [27] which state that cross validation is not much worse than a holdout set and some slightly stronger results [?] and [25].

PROBLEM 4.1.5.

(Open) Construct a bound on the deviation of cross validation for arbitrary classifiers which is a quantitative improvement on the results of [?].

[next] [front] [up]