8.4 Shell Bounds for Continuous Spaces

Applying Shell bounds to continuous hypothesis spaces is not easy. In fact, upon first inspection, this appears to be impossible since shell bounds require knowledge of the number of hypotheses with a particular empirical error. We can avoid these difficulties, while introducing a small amount of slack, by always being concerned with the measure rather than the count of hypotheses. In particular, we will keep track of the measure of hypotheses with a confusingly small error and the measure of the hypotheses that we pick. The approach here is similar to the approach in section 6 although more simplistic.

First assume that there is some measure

Q

over the hypothesis space

H

. Suppose that we choose some subset,

U

, of the hypotheses with measure

Q (U)

. We will be concerned with the largest empirical error rate

{\hat{e}}_{U} (h) = {max}_{h \in U} \hat{e} (h)

and the average true error rate,

e_{U} (h) = E_{Q_{U}} e (h)

. A bound on the gap between the largest empirical error rate and the average true error rate will imply a true error bound for the stochastic classifier which chooses randomly and evaluates.

P_{s} (ε, K) \equiv \int_{h : e (h) > K + ε} Q (h) Bin (K, e (h), m) d h

We will also need the concept of rounding. Choose

i

such that

\frac{1}{m^{i}} \geq Q (U) \geq \frac{1}{m^{i + 1}}

then, define:

\delimiter Q (U)\delimiter  = \frac{1}{m^{i + 1}}

Now, the following theorem holds:

THEOREM 8.4.1.

(Stochastic Full knowledge) For all distributions $Q$ , For all $δ \in (0, 1]$ : ${Pr}_{D^{m}} (\exists U e_{U} (h) \leq {\hat{e}}_{U} (h) + \frac{1}{m} + ε (δ, \hat{e}, U))$ where $ε (δ, \frac{K}{m}, U) = min \{ε : P_{s} (ε, K) \leq \frac{δ \delimiter Q (U)\delimiter }{m^{3}}\}$

PROOF. Call a hypothesis with a large true error ( $e (h) > \frac{K}{m} + ε$ ) and small empirical error ( $\hat{e} (h) \leq \frac{K}{m}$ ) a “bad” hypothesis. $P_{s} (ε, K)$ is the expected measure of the bad hypotheses. We will use Markov’s inequality to bound the actual measure of bad hypotheses. Then, given that the quantity is bounded, we can bound the expected true error by assuming that we included every bad hypothesis in the set $U$ .

Let $ε_{K i} = min \{ε : P_{s} (ε, K) \leq \frac{δ}{m^{3 + i}}\}$ Intuitively, $ε_{K i}$ is the value we will use when $\hat{e} = \frac{K}{m}$ and $\delimiter Q (\hat{e})\delimiter = \frac{1}{m^{i}}$ . Also let ${\hat{P}}_{s} (ε, K) = \int_{h : e (h) > K + ε \land \hat{e} (h) \leq K} p (h) d h$ be the actual measure of bad hypotheses. Then, Markov’s inequality tells us: $\forall δ > 0 {Pr}_{D} ({\hat{P}}_{s} (ε_{K i}, K) \geq \frac{m^{2}}{δ} P_{s} (ε_{K i}, K)) \leq \frac{δ}{m^{2}}$ Taking the union bound over all values of $ε_{K i}$ , we get: $\forall δ > 0 {Pr}_{D} (\forall K, i {\hat{P}}_{s} (ε_{K i}, K) \geq \frac{m^{2}}{δ} P_{s} (ε_{K i}, K)) \leq δ$ So, with probability $1 - δ$ for all values of $K$ and $i$ , we have: ${\hat{P}}_{s} (ε_{K i}, K) \leq \frac{1}{m^{1 + i}}$ . Therefore, if $Q (U) \geq \frac{1}{m^{i}}$ , we know that $\frac{Q (U)}{{\hat{P}}_{s} (ε_{K i}, K)} \geq m$ . Assuming that all of the bad hypotheses have true error $1$ and all the rest have true error at most $\frac{K}{m} + ε_{K i}$ , we get the following true error bound: $e_{U} (h) \leq {\hat{e}}_{U} (h) + \frac{1}{m} + ε_{\hat{e} i}$ ▫

The stochastic full knowledge bound is loose when applied to the full knowledge setting by

δ \to \frac{δ}{m^{2}}

. Typically, this results in only a

\frac{2 ln m}{m}

increase in the size of

ε

, although the increase can sometimes be much larger near phase transitions. These factors of

\frac{ln m}{m}

may be removable with improved argumentation. Naturally, the stochastic full knowledge bound can be used to prove a stochastic observable bound.

The next theorem is the observable analog of the stochastic full knowledge bound. Here, we eliminate the unobservable quantities to produce a stochastic observable shell bound. The observable quantity we will use is:

{\hat{P}}_{s} (ε, K, δ) \equiv 2 \sum_{h} Q (h) Bin (K, \bar{e} (ε, K, δ, \hat{e} (h)), m)

where

\bar{e} (ε, K, δ, \frac{i}{m}) = max \{K + ε, min \{p : 1 - Bin (m, K, p) \geq δ\}\}

is a slightly pessimal estimate of the true error rate given the empirical error rate as before.

THEOREM 8.4.2.

(Stochastic Observable Shell Bound) For all distributions $Q$ , For all $δ \in (0, 1]$ : ${Pr}_{D^{m}} (\exists U e_{U} (h) \leq {\hat{e}}_{U} (h) + \frac{1}{m} + ε (δ, \hat{e}, U))$ where $ε (δ, \frac{K}{m}, U) = min \{ε : {\hat{P}}_{s} (ε, K, \frac{δ}{2}) \leq \frac{δ \delimiter Q (U)\delimiter }{2 m^{3}}\}$

PROOF. First note that the unobservable bound lemma 8.1.3 implies that with probability $1 - \frac{δ}{2}$ , we have ${\hat{P}}_{s} (ε, K, \frac{δ}{2}) \geq P_{s} (ε, K)$ . Given that this is the case, our choice of $ε$ will be at least as pessimistic as the choice defined by the Stochastic Full Knowledge bound 8.4.1. We thus have two sources of failure: the unobservable bound lemma will fail with probability at most $\frac{δ}{2}$ and the stochastic full knowledge bound will fail with probability $\frac{δ}{2}$ . The union bound then implies that the Stochastic Observable Shell Bound holds with probability $1 - \frac{δ}{2}$ . ▫

The information requirements for the continuous space shell bound are even more severe than the requirements for the discrete space shell bound. In particular, we need to know the exact measure of the hypotheses with any particular empirical error. Clearly, this is unrealistic. We can relax this requirement to observations computable in a finite amount of time using the sampling techniques of theorem 8.2.1. In particular, given a well-behaved distribution

Q

, we can make a bounded estimate of the measure of hypotheses with some empirical error rate. Given this bounded estimate, we can then calculate a pessimistic shell bound for the continuous space.

8.4. Shell Bounds for Continuous Spaces