8.1 The Discrete Shell Bound

Let

B (m, K, p) = (\binom{m}{K}) e {(h)}^{K} {(1 - e (h))}^{m - K}

be the probability that hypothesis with true error

p

produces

K

errors on

m

independent examples.

The discrete shell bound works directly with the probability that there will be a confusingly small empirical error. Let

P (ε, K) \equiv \sum_{h : e (h) > K + ε} B (m, K, e (h))

Intuitively,

P (ε, K)

is a bound on the probability that a hypotheses with a true error rate larger than

\frac{K}{m} + ε

will have an empirical error rate of

\frac{K}{m}

. The contribution to the sum will fall off exponentially as the true error,

e (h)

, increases.

Our first step is stating a shell bound which requires unknown information. The purpose of this bound is motivational - it provides incite into why we can expect a large improvement. Later, we will remove the unknown information requirements and recover a useful bound.

THEOREM 8.1.1.

(Full knowledge theorem) For all $δ \in (0, 1]$ : ${Pr}_{D^{m}} (\exists h \in H e (h) \geq \hat{e} (h) + ε (δ, \hat{e} (h))) \leq δ$ where $ε (δ, \frac{K}{m}) = min \{ε : P (ε, K) = \frac{δ}{m}\}$

The full knowledge theorem relies on unobservable information—the true error rates of all hypotheses. This theorem is not (quite) trivial because it does not rely upon information about which hypothesis has a particular true error.

PROOF. For every hypothesis with a true error rate of $e (h) > \frac{K}{m} + ε (δ, K)$ the probability of producing an empirical error of $\hat{e} (h) = \frac{K}{m}$ is $B (K, e (h), m)$ . Applying the union bound over all hypotheses and all $m$ possible nontrivial values of $K$ completes the proof. ▫

There are a couple things to note about this theorem. First, for most balanced machine learning problems most hypotheses typically have a true error rate near to

0.5

. Given this, and noticing that Binomial tails fall of exponentially, dramatic improvements in the bound are feasible.

Second, we must use

\frac{δ}{m}

rather than

δ

in order to make the proof work. It is possible that theorem holds without splitting

δ

“

m

-ways”. Removing the factor of

m

is an open problem. For the special case of empirical risk minimization algorithms, McDiarmid’s inequality [40] implies that the range of hypotheses with minimum empirical error is of size

O (\sqrt{m})

with high probability. Therefore, we need only apply the union bound to

O (\sqrt{m})

possible minimum empirical error rates.

8.1.2. No knowledge of learning distribution

Applying the full knowledge theorem ( 8.1.1) is not practical in almost all learning settings because we do not know the distribution of true error rates. Therefore, it is necessary to construct a slightly looser theorem which relies upon only observable quantities. Surprisingly, this is possible while introducing only slightly more slack.

\bar{e} (ε, K, δ, \frac{i}{m}) = max \{\frac{K}{m} + ε, min \{p : 1 - Bin (m, i, p) \geq δ\}\}

Intuitively,

\bar{e} (ε, K, δ, \frac{i}{m})

is a lower bound on the true error rate of the hypothesis with an empirical error of

\frac{i}{m}

\hat{P} (ε, K, δ) \equiv 2 \sum_{h} B (m, K, \bar{e} (ε, K, δ, \hat{e} (h)))

The quantity

\hat{P} (ε, K, δ)

is an upper bound on the probability that one of the hypotheses with a true error rate of

\bar{e}

or more could produce

K

empirical errors.

Noting that there are only

m + 1

possible empirical errors, we can first let

c (\frac{i}{m}) = |\{h : \hat{e} (h) = \frac{i}{m}\}|

be the count of empirical errors at

\frac{i}{m}

. Then we can redefine:

\hat{P} (ε, K, δ) \equiv 2 \sum_{i = 0}^{m} c (\frac{i}{m}) B (m, K, \bar{e} (ε, K, δ, \frac{i}{m}))

Later, we will prove that with high probability,

\hat{P} (ε, K, δ) \geq P (ε, K)

. Given that this is so, we can prove a theorem which only relies on observable quantities.

THEOREM 8.1.2.

(Observable Shell Bound) For all $δ > 0$ : ${Pr}_{D^{m}} (\exists h \in H e (h) \geq \hat{e} (h) + ε (δ, \hat{e} (h))) \leq δ$ where $ε (δ, \frac{K}{m}) = min \{ε : \hat{P} (ε, K, \frac{δ}{2}) \leq \frac{δ}{2 m}\}$

The observable shell bound preserves the important locality property of the full knowledge shell bound. In particular, when most of the true error rates are “far” from the empirical error rate (and

m

is large enough), we expect to make large (functional) improvements on the discrete hypothesis bound 4.2.1.

The proof rests upon a technical lemma which allows us to bound the unobservable “probability of a misleading event” with an observable event.

LEMMA 8.1.3.

(Unobservable bound) For all empirical errors, $K$ , for all distributions $Q (h)$ , for all $δ \in (0, 1]$ : ${Pr}_{D^{m}} (2 \sum_{h} Q (h) B (m, K, \bar{e} (ε, K, δ, \hat{e} (h))) \geq \sum_{h : e (h) > K + ε} Q (h) B (m, K, e (h))) \leq δ$

This lemma is powerful because it bounds the unobservable right hand side in terms of the observable left hand side.

PROOF. Let $\bar{e} (m, \frac{k}{m}, δ) \equiv {min}_{p} {p : 1 - Bin (m, k, p) \geq δ}$ $\forall α \in (0, 1] \forall h : E_{D^{m}} I (e (h) < \bar{e} (m, \hat{e} (h), α)) \leq α$ $\Rightarrow \forall P (h) E_{P} E_{D^{m}} I (e (h) < \bar{e} (m, \hat{e} (h), α)) \leq α$ $\Rightarrow \forall P (h) E_{D^{m}} E_{P} I (e (h) < \bar{e} (m, \hat{e} (h), α)) \leq α$ $\Rightarrow \forall P (h) {Pr}_{D^{m}} (E_{P} I (e (h) < \bar{e} (m, \hat{e} (h), α)) \geq \frac{α}{δ}) \leq δ$ $\Rightarrow \forall P (h) {Pr}_{D^{m}} ({Pr}_{P} (e (h) < \bar{e} (m, \hat{e} (h), α)) \geq \frac{α}{δ}) \leq δ$ Let $P (h) \propto Q (h) B (m, K, e (h))$ . Then: $\Rightarrow \forall Q (h) {Pr}_{D^{m}} (\frac{\sum_{h : e (h) > \frac{K}{m} + ε \land e (h) > \bar{e} (m, \hat{e} (h), α)} Q (h) B (m, K, e (h))}{\sum_{h : e (h) > \frac{K}{m} + ε} Q (h) B (m, K, e (h))} \geq \frac{α}{δ}) \leq δ$ set $α = \frac{δ}{2}$ and replace $e (h)$ with $\bar{e} (m, \hat{e} (h), α)$ to achieve the result. ▫

PROOF. (of theorem 8.1.2) Choose $Q (h)$ = the uniform distribution on a our hypothesis space. Then, we know that with probability $1 - δ$ , $\hat{P} (ε, K, δ) \geq P (ε, K)$ . Therefore, we can (arbitrarily) allocate a $\frac{δ}{2}$ probability of failure to the unobservable bound 8.1.3 and a $\frac{δ}{2}$ probability of failure to the full knowledge bound 8.1.1. Assuming $\hat{P} (ε, K, δ) \geq P (ε, K)$ , the observable bound will be more pessimistic than the full knowledge bound. ▫

The Observable Shell bound behaves in a strange manner which is unlike other true error bounds. In particular, the true error bound can be discontinuous in the value of

δ

. This discontinuity implies that relatively small improvements in the shell bounds can result in dramatic improvements in the value of the true error bound. While dramatic improvements can happen with small improvements in the shell bound, we expect that in practice, such large improvements will not be too common, simply because a small improvement is unlikely (amongst all learning problems) to shift the bound across one of these discontinuous transitions.

8.1. The Discrete Shell Bound

8.1.1. Knowledge of learning distribution

8.1.2. No knowledge of learning distribution