8.3 Lower Bounds

The lower upper bound 4.4 does not apply to shell bounds because we are using more information than just the empirical error rate of a learned hypothesis. In particular, we are using the empirical error rates of all the hypotheses in calculating the bound. Is there a lower upper bound which applies for the information used by the shell bound? The same independent hypothesis technique will allow us to lower bound the full knowledge theorem 8.1.1. In particular, assume that we are given a set of independent hypotheses, each with some true error

e (h)

. What is a lower bound on the probability that one of these hypotheses will have an empirical error of

\frac{K}{m}

A

and

B

are independent events, then:

Pr (A or B) = Pr (A) + Pr (B) - Pr (A and B)

= Pr (A) + Pr (B) - Pr (A) Pr (B)

= (1 - Pr (B)) Pr (A) + Pr (B)

This implies that we can “add” the independent probabilities together as long as we rescale. In particular,

B

might be the probability of a “bad” hypothesis in some set of hypotheses and

A

might be the probability that some new hypothesis with a large true error rate has a small empirical error.

THEOREM 8.3.1.

(Lower Upper Shell Bound) For all true errors $e (h)$ , $K$ : ${Pr}_{D^{m}} (\exists h : e (h) > \frac{K}{m} + ε and \hat{e} (h) = \frac{K}{m}) \geq P (ε, K) (1 - P (ε, K))$

PROOF. The proof is by finite induction on the set of hypotheses with a large true error rate. Let $P_{H} (ε, K) = \sum_{h \in H} B (m, K, e (h))$ be the sum of the probabilities that each hypothesis in $H^{'}$ produces an empirical error of $\frac{K}{m}$ . Now, we want to prove that: $\forall H {Pr}_{D^{m}} (\exists h \in H : \hat{e} (h) = \frac{K}{m}) \geq P_{H} (ε, K) (1 - P_{H} (ε, K)$ This is true for the base case of $∣ H ∣ = 1$ . Assuming that it is true for the case of $∣ H ∣ = n$ , we need to prove it for the case of $H^{'} = H \cup ∣ h ∣$ . In particular, we have assumed ${Pr}_{D^{m}} (\exists h \in H : \hat{e} (h) = \frac{K}{m}) \geq P_{H} (ε, K) (1 - P_{H} (ε, K)$ Using the earlier independent principle, we get that: ${Pr}_{D^{m}} (\exists h \in H^{'} : \hat{e} (h) = \frac{K}{m})$ $\geq P_{H} (ε, K) (1 - P_{H} (ε, K)) + (1 - P_{H} (ε, K) (1 - P_{H} (ε, K))) B (K, e (h), m)$ $\geq P_{H} (ε, K) (1 - P_{H} (ε, K)) + (1 - P_{H} (ε, K)) B (K, e (h), m)$ $= P_{H^{'}} (ε, K) (1 - P_{H} (ε, K))$ $\geq P_{H^{'}} (ε, K) (1 - P_{H^{'}} (ε, K))$ By induction, this property therefore holds for the set of all hypotheses with a large true error rate. ▫

Assuming that

δ < \frac{1}{2}

, the lower upper shell bound is tight to within a factor (in

δ

) of

2 m

with the full knowledge bound 8.1.1. Given the exponential behavior of Binomial tails, this usually (but not always) implies a small impact on the true error bound. One important question remains: how does this bound compare to the observable shell bound 8.1.2? In the observable shell bound, the distribution of true errors is replaced with a pessimistic distribution based upon the observed empirical errors. The “size” of this pessimism in terms of the true error bound is, in general, of size

\frac{1}{\sqrt{m}}

. Thus, the gap between the lower upper shell bound and the upper shell bound is typically of size

\frac{1}{\sqrt{m}}

8.3. Lower Bounds