4.4 Lower Upper Bounds

The second form of lower bound is a lower bound on the upper bound (similar to the results of [14]). If we can lower bound the upper bound (as a function of its observables), then we can be confident that the upper bound is no looser than the gap between the lower upper bound and the upper bound.

How tight is the discrete hypothesis bound ( 4.2.1)? The answer is sometimes tight. In particular, we can exhibit a set of learning problem where the discrete hypotheses bound can not be made significantly lower as a function of the observables,

m

δ

∣ H ∣

, and

\hat{e} (h)

. Fix the value of these quantities and then we will construct a learning problem for which a lower upper bound can not be stated.

Our learning problem will be defined over the input space

X = {0, 1}^{∣ H ∣}

. The hypotheses will be

h_{i} (x) = x_{i}

where

x_{i}

is the

i

th component of the vector

x

. This construction allows us to vary the true error rate of each hypothesis independent of the other hypotheses. In fact, we can pick any true error rate for any hypothesis by simply adjusting the probability that

x_{i} = y

. Our learning problem can therefore generate problems according to the following algorithm:

By construction, the true error rate of each hypothesis will be

e (h_{i})

. Now, we can prove the following theorem:

THEOREM 4.4.2.

(Discrete Hypothesis lower upper bound) For all true error rates $e (h) > \frac{ln ∣ H ∣ + ln \frac{1}{δ}}{m}$ there exists a learning problem and algorithm such that: ${Pr}_{D^{m}} (\exists h \in H : e (h) \geq \bar{e} (m, \hat{e} (h), \frac{δ}{∣ H ∣})) \geq 1 - e^{- δ}$ where $\bar{e} (m, \frac{k}{m}, δ) \equiv {max}_{p} {p : Bin (m, k, p) = δ}$

Intuitively, this theorem implies that we can not improve significantly on the results of theorem 4.2.1 without using extra information about our learning problem. Some of our later results do exactly this - they use extra information.

PROOF. Using the family of learning problems implicitly defined by algorithm 4.4.1, we know that $\forall h, \forall δ \in [0, 1] : {Pr}_{D^{m}} (e (h) \geq \bar{e} (m, \hat{e} (h), \frac{δ}{∣ H ∣})) \geq \frac{δ}{∣ H ∣}$ (negation) $\Rightarrow \forall h, \forall δ \in [0, 1] : {Pr}_{D^{m}} (e (h) < \bar{e} (m, \hat{e} (h), \frac{δ}{∣ H ∣})) < 1 - \frac{δ}{∣ H ∣}$ (independence) $\Rightarrow \forall δ \in [0, 1] : {Pr}_{D^{m}} (\forall h e (h) < \bar{e} (m, \hat{e} (h), \frac{δ}{∣ H ∣})) < {(1 - \frac{δ}{∣ H ∣})}^{∣ H ∣}$ (negation) $\Rightarrow \forall δ \in [0, 1] : {Pr}_{D^{m}} (\exists h e (h) \geq \bar{e} (m, \hat{e} (h), \frac{δ}{∣ H ∣})) \geq 1 - {(1 - \frac{δ}{∣ H ∣})}^{∣ H ∣}$ (approximation) $\Rightarrow \forall δ \in [0, 1] : {Pr}_{D^{m}} (\exists h e (h) \geq \bar{e} (m, \hat{e} (h), \frac{δ}{∣ H ∣})) \geq 1 - e^{- δ}$ ▫

For small

δ

, we get that

1 - e^{- δ} ≃ δ

which implies that, no significantly better true error bound can be stated for all learning algorithms. In particular, the empirical risk minimization algorithm (which chooses the hypothesis with minimal empirical error) will have a significant probability of a large deviation between the empirical and true error.

4.4. Lower Upper Bounds