9.3 Bracketing Covering Number Bound

There is an alternative version of the covering number bounds which has been little used in learning theory. This is mentioned as the “direct method” in [20] and Section II.2 of [43] discusses this approach but is only concerned with asymptotic consistency rather than rates of convergence.

We start with a more restricted notion of covering—a covering in which we bracket the hypotheses. Let

Z

be the space of

(x, y)

labeled examples and

h (Z) = {(x, y) : h (x) \neq y}

in:

N_{[]} (H, γ, d_{D}) = {inf}_{F} |F : \forall h \in H \exists (f, f^{'}) \in F : \begin{matrix} d_{D} (f, f^{'}) \leq γ \\ and f (Z) > h (Z) > f^{'} (Z) \end{matrix}|

In other words,

f

and

f^{'}

are two sets which are “above” and “below” every hypothesis that they cover. Note that it is very important in this definition that the sets

f

and

f^{'}

are not required to correspond to functions in

H

. In fact, they can simply be understood as arbitrary sets of

(x, y)

pairs.

It is immediately obvious that:

N_{[]} (H, γ, d_{D}) \geq N (H, γ, d_{D})

However, the relationship between

N_{[]} (H, γ, d_{D})

and

C (H, γ)

is ambiguous: either could be larger.

THEOREM 9.3.1.

Let $f (h)$ be the upper bracket of any hypothesis, $h$ . For all $γ \in (0, 1]$ , for all $δ \in (0, 1]$ : ${Pr}_{D^{m}} (\exists h \in H : \begin{matrix} e (h) \geq \bar{e} (m, \hat{e} (f (h)), \frac{δ}{2 N_{[]} (H, γ, d_{D})}) \\ or \hat{e} (f (h)) - \hat{e} (h) \geq b (m, γ, \frac{δ}{2 N_{[]} (H, γ, d_{D})}) \end{matrix}) \leq δ$ where $\begin{matrix} \bar{e} (m, \frac{K}{m}, δ) \equiv {max}_{p} {p : Bin (m, K, p) = δ} \\ and b (m, p, δ) \equiv^{'} {min}_{K} \{\frac{K}{m} : 1 - Bin (m, K, p) \leq δ\} \end{matrix}$ .

This theorem constrains the distance between

\hat{e} (f (h))

and

e (h)

and the distance between

\hat{e} (f (h))

and

\hat{e} (h)

. Consequently, it constrains the distance between

e (h)

and

\hat{e} (h)

. The proof of the theorem rests on the following two lemmas which each assert a convergence. Graphically, the proof has the following form:

LEMMA 9.3.2.

Let $f (h)$ be the upper bracket of any hypothesis, $h$ . For all $γ \in (0, 1]$ , for all $δ \in (0, 1]$ : ${Pr}_{D^{m}} (\exists h \in H : \hat{e} (f (h)) - \hat{e} (h) \geq b (m, γ, \frac{δ}{N_{[]} (H, γ, d_{D})})) \leq δ$ where $b (m, p, δ) \equiv^{'} {min}_{K} \{\frac{K}{m} : 1 - Bin (m, K, p) \leq δ\}$

PROOF. If $f (h), f^{'} (h)$ bracket $h$ , we have: $e (f (h)) - e (f^{'} (h)) \leq γ$ Therefore a coin which is “heads” when $f (h) (z) \neq f^{'} (h) (z)$ has a bias of $γ$ or less. Since $f^{'} (h)$ is correct everywhere that $h$ is correct, we also know: $\forall ε {Pr}_{D^{m}} (\hat{e} (f (h)) - \hat{e} (h) \geq ε) \leq {Pr}_{D^{m}} (\hat{e} (f (h)) - \hat{e} (f^{'} (h)) \geq ε)$ Therefore, we have at most $N_{[]} (H, γ, d_{D})$ Binomial distributions, each with bias at most $γ$ , and wish to bound the probability of a large deviation. Using an upper binomial tail calculation, we get the result. ▫

LEMMA 9.3.3.

For all $δ \in (0, 1]$ : ${Pr}_{D^{m}} (\exists f, f^{'} \in N_{[]} (H, γ, d_{D}) : e (f (h)) \geq \bar{e} (m, \hat{e} (f (h)), \frac{δ}{N_{[]} (H, γ, d_{D})}))$ where $\bar{e} (m, \frac{K}{m}, δ) \equiv {max}_{p} {p : Bin (m, K, p) = δ}$

PROOF. Application of the discrete hypothesis space bound 4.2.1 for $f$ in every $(f, f^{'})$ pair. ▫

PROOF. (of theorem 9.3.1) Allocate $\frac{δ}{2}$ confidence to each lemma and use the union bound with both lemmas to get: ${Pr}_{D^{m}} (\exists h \in H : \begin{matrix} e (f (h)) \geq \bar{e} (m, \hat{e} (f (h)), \frac{δ}{2 N_{[]} (H, γ, d_{D})}) \\ \hat{e} (f (h)) - \hat{e} (h) \geq b (m, γ, \frac{δ}{2 N_{[]} (H, γ, d_{D})}) \end{matrix}) \leq δ$ where $\begin{matrix} \bar{e} (m, \frac{K}{m}, δ) \equiv {max}_{p} {p : Bin (m, K, p) = δ} \\ and b (m, p, δ) \equiv^{'} {min}_{K} \{\frac{K}{m} : 1 - Bin (m, K, p) \leq δ\} \end{matrix}$ . Since $e (h) \leq e (f (h))$ by construction, the proof is complete. ▫

The alternative covering number argument has a significant advantage over the standard argument: when restricted to a finite hypothesis space, the argument becomes tight. In particular, on a finite hypothesis space, we can set

γ = 0

and get:

N_{[]} (H, 0, d_{D}) \leq ∣ H ∣

. The bound reduces to the standard discrete hypothesis space bound 4.2.1. Consequently, there exist learning problems where this bound is quite tight.

The bracketing covering approach has the following advantages and disadvantages:

In a sense, this bound is useless because it requires knowledge of the unknown distribution

D

in order to calculate a covering number. In the next section, we will see that bounds on the bracketing covering number which hold for all

D

can often be found.

9.3. Bracketing Covering Number Bound