8.2 Sampling Shell Bound

The Shell bound relies upon the distribution of empirical errors across the entire hypothesis space. Calculating

c (\frac{i}{m})

, while theoretically possible, is often practically intractable. For example, the space of all binary functions on

n

features has size

2^{2^{n}}

and for a decision tree with a number of nodes

k = O (m)

there are more than

2^{k}

hypotheses with the same number of nodes. In order to avoid this difficulty, we will use sampling which is made possible by noticing that the Shell bound does not require exact knowledge of the empirical error distribution. Instead, we can safely count a hypothesis twice because over-counting monotonically worsens the bound. Assume that we have an oracle which can be used to sample uniformly from the set of all hypotheses. Then, we can bin the sampled hypotheses according to their error rate on the example set. After repeating

k

times we will arrive at an empirical distribution over error rates which can be altered into a bound on the true distribution of error rates by upper bounding the count

c (\frac{i}{m})

. Let

\hat{c} (\frac{i}{m})

be the observed number of hypotheses out of

k

uniform choices with empirical error

\frac{i}{m}

and define:

\bar{c} (k, \frac{\hat{c}}{k}, δ, ∣ H ∣) \equiv ∣ H ∣ * {max}_{p} \{p : Bin (k, \hat{c}, p) = \frac{δ}{m}\}

Intuitively,

\bar{c}

will be a high confidence upper bound on the number of hypotheses with empirical error rate

\hat{c}

. Given these quantities, we can calculate an approximate

\hat{P}

according to the formula:

\hat{\hat{P}} (ε, K, δ) \equiv 2 \sum_{i = 0}^{m} \bar{c} (k, \frac{\hat{c} (\frac{i}{m})}{k}, \frac{δ}{2 m}, ∣ H ∣) B (m, K, \bar{e} (ε, K, \frac{δ}{2}, \frac{i}{m}))

THEOREM 8.2.1.

(Sampling Shell Bound) For all $δ > 0$ : ${Pr}_{D^{m}} (\exists h \in H e (h) \leq \hat{e} (h) + ε (δ, \hat{e} (h))) \leq δ$ where $ε (δ, K) = min \{ε : \hat{\hat{P}} (ε, K, \frac{δ}{2}) \leq \frac{δ}{2 m}\}$

PROOF. For every $i$ , we know that $\hat{\hat{P}} (ε, K, δ) \geq \hat{P} (ε, K, \frac{δ}{2})$ with probability at least $1 - \frac{δ}{2}$ . This implies that $\hat{\hat{P}} (ε, K, δ) \geq P (ε, K)$ with probability at least $1 - δ$ . Applying the union bound with the full knowledge theorem gives us the result. ▫

The Sampling Shell bound is still relatively fast to calculate given the sampling results, but it is worth noting that many samples are required—the bound will typically tighten linearly with

ln k

. In other words, an exponentially large

k

is required to realize all of the gains of the shell bound. Thus, the sampling shell bound will interpolate between the discrete hypothesis bound 4.2.1 and the shell bound as

ln k

moves from

1

ln ∣ H ∣

8.2. Sampling Shell Bound