9.4 Covering number calculations

It is important to demonstrate that this covering number is feasible to calculate and gives a better answer than the traditional approach. We will do this by first calculating the bracketing covering number for a very simple continuous classifier and then comparing the results with the traditional covering number approach.

Bracketing covering numbers have already been proved for many function classes [13][49]. Here, we will present a proof for the simplest of continuous hypothesis spaces: the step function on a line segment. Each hypothesis will be indexed by a number

a \in [0, 1]

according to:

h_{a} (x) = sign (x - a)

What is

N_{[]} (H, γ, d_{D})

for this hypothesis set?

THEOREM 9.4.1.

Assume that $D$ can be described by a probability density function, then:

$N_{[]} (H, γ, d_{D}) \leq \frac{1}{γ} + 1$

PROOF. Consider a range of hypotheses from $h_{a}$ to $h_{b}$ . For this range of hypotheses, we can choose a bracketing pair, $(f, f^{'})$ . In particular, we can choose $f$ and $f^{'}$ which agree on $[0, a)$ and $[b, 1]$ and always predict either incorrectly ( $f$ ) or correctly ( $f^{'}$ ) on $[a, b)$ . The distance between these functions satisfies: $d_{D} (f, f^{'}) = {Pr}_{D} (x \in [a, b))$ and every hypothesis $h_{c}$ in $[a, b)$ satisfies $\forall z f (z) \geq h_{c} (z) \geq f^{'} (z)$ . Consequently, if $a$ and $b$ are chosen appropriately, we will observe $d_{D} (f, f^{'}) \leq γ$ .

If $D$ can be described by a probability distribution, then we can simply calculate the marginal distribution, $D_{x}$ , and the cumulative distribution of the margin, $F_{D} (x)$ . Now, find $a_{i}$ for which $F_{D} (a_{i}) = \frac{i}{γ}$ for $i < \frac{1}{γ}$ . Choose $b_{i} = a_{i + 1}$ . There are at most $\frac{1}{γ} + 1$ intervals, each with a measure (according to $D$ ) of at most $γ$ . Consequently, we can cover $H$ with $\frac{1}{γ} + 1$ pairs of $(f_{i}, f_{i}^{'})$ . ▫

Given that the bracketing cover is

\frac{1}{γ} + 1

, we can use theorem 9.3.1 to define a constraint that the true error rate must satisfy with high probability. Setting

γ = \frac{1}{m - 1}

, we get:

\hat{e} (f (h)) \leq \hat{e} (h) + b (m, \frac{1}{m - 1}, \frac{δ}{2 m})

and

e (h) \leq \bar{e} (m, \hat{e} (f (h)), \frac{δ}{2 m})

To be fair in comparison to the standard covering number approaches, we should relax our theorem to use the Hoeffding approximation. Note that this is a bit unfair because the first inequality is (inherently) a highly biased Binomial with lower variance. Relaxing to the Hoeffding bound, we get:

\hat{e} (f (h)) \leq \hat{e} (h) + \frac{1}{m - 1} + \sqrt{\frac{ln \frac{2 m}{δ}}{2 m}}

and

e (h) \leq \hat{e} (f (h)) + \sqrt{\frac{ln \frac{2 m}{δ}}{2 m}}

which implies:

e (h) \leq \hat{e} (h) + \frac{1}{m - 1} + 2 \sqrt{\frac{ln 2 m + ln \frac{1}{δ}}{2 m}}

Note again that we are being “unfair” to the new approach by using the Hoeffding approximation rather than exact Binomial-tail bounds. The standard covering number approach has not yet been reduced to exact Binomial-tail bounds. Using the standard approach, the covering number, we get

C (H, \frac{1}{m - 1}) = m

. This implies a bound of:

e (h) \leq \hat{e} (h) + 4 \sqrt{\frac{ln 8 m + ln \frac{1}{δ}}{m}}

Comparing the bounds, we see that the new approach is about

\frac{16}{2} = 8

times more efficient in the number of examples required to achieve a bound on a given deviation.

9.4.1. Note on the Bracketing Cover proof

In fact, the proof can be extended to all

D

(even ones with point masses) at the cost of a factor of

2

worsening and a messier argument. Property (2) is desirable because it is often not the case that we know the distribution

D

when we wish to apply the bound. Property (1) is an essential technique that can be used to prove other covering number bounds for this notion of covering number.

Can we show partial order covering number bounds for other classifiers? There is a straightforward extension of the previous proof for classifiers which consist of axis parallel intervals in

R^{n}

. More work is required to prove partial order covering numbers for the hypothesis spaces of standard learning algorithms.

9.4. Covering number calculations

9.4.1. Note on the Bracketing Cover proof