4.2 The basic training set bound

The most basic of training set based sample complexity bounds is the simple combination of a Binomial tail bound and the union bound. In particular, we have:

THEOREM 4.2.1.

(Discrete Hypothesis Bound) For all hypothesis spaces, $H$ , for all $δ \in (0, 1]$ ${Pr}_{D^{m}} (\exists h \in H : e (h) \geq \bar{e} (m, \hat{e} (h), \frac{δ}{∣ H ∣})) \leq δ$ where $\bar{e} (m, \frac{k}{m}, δ) \equiv {max}_{p} {p : Bin (m, k, p) = δ}$

Note that this theorem can only be nonvacuous when the hypothesis space,

H

, has some finite (discrete) size.

PROOF. For every individual hypothesis, we know that: $\forall h {Pr}_{D^{m}} (e (h) \geq \bar{e} (m, \hat{e} (h), \frac{δ}{∣ H ∣})) \leq \frac{δ}{∣ H ∣}$ Applying the union bound (see section 3.5) for every hypothesis gives us: ${Pr}_{D^{m}} (\exists h \in H : e (h) \geq \bar{e} (m, \hat{e} (h), \frac{δ}{∣ H ∣})) \leq δ$ which is the result. ▫

Intuitively, this theorem says that as the number of hypotheses grows, we can not guarantee that the empirical error will be near to the true error.

A better understanding can be gained by considering some of the Binomial tail bound approximations. This is also worth mentioning in order to compare this theorem with theorems in more common forms.

COROLLARY 4.2.2.

(Relative Entropy Discrete Hypothesis Bound) For all hypothesis spaces, $H$ , for all $δ \in (0, 1]$ : ${Pr}_{D^{m}} (\exists h \in H : KL (\hat{e} (h) ∣ ∣ e (h)) \geq \frac{ln ∣ H ∣ + ln \frac{1}{δ}}{m}) \leq δ$

PROOF. Loosen (theorem 4.2.1) with the relative entropy Chernoff bound (Eqn. 3.2.1), and use the inversion lemma 3.4.1. ▫

To understand this lemma, it is helpful to consider some approximations of

KL (\hat{e} (h) ∣ ∣ e (h))

. In particular, we have:

∣ \hat{e} (h) - e (h) ∣ \geq KL (\hat{e} (h) ∣ ∣ e (h)) \geq 2 {(\hat{e} (h) - e (h))}^{2}

which implies that the KL-divergence varies between an

l_{1}

and an

l_{2}

metric.

We can further loosen the last corollary with the Hoeffding approximation to get the following commonly stated bound:

COROLLARY 4.2.3.

(Agnostic Discrete Hypothesis Bound) For all hypothesis spaces, $H$ , for all $δ > 0$ , ${Pr}_{D^{m}} (\exists h \in H : e (h) \geq \hat{e} (h) + \sqrt{\frac{ln ∣ H ∣ + ln \frac{1}{δ}}{2 m}}) \leq δ$

PROOF. Loosen corollary ( 4.2.2) with the bound $KL (q ∣ ∣ p) \geq 2 {(q - p)}^{2}$ . ▫

The agnostic form of this bound has the advantage that it explicitly shows that we are forcing the convergence (in high probability) of the empirical error rate to the true error rate. Graphically, we are forcing every empirical error to be near to its true error. This is a picture which represents the connection

Later bounds will have more complicated convergence conditions with more complicated graphs. These more complicated bounds are necessary in order to avoid the limitations of the holdout bound. See figure 12.3.3 for a comparison with the holdout set.

4.2. The basic training set bound