3.2 Approximation techniques

3.2. Approximation techniques

Exact calculation of $Bin (m, k, p)$ (covered in the next subsection) can require computation at least proportional to $m$ , which is often too expensive. For the bounds in this thesis, we will only need to calculate an upper bound on the quantity $Bin (m, k, p)$ . There are several inequalities which are often used. The first of these is the Hoeffding inequality[23]. Assume that $\frac{k}{m} < p$ then we have: $Bin (m, k, p) \leq e^{- 2 m {(p - \frac{k}{m})}^{2}}$ Intuitively, this inequality can be seen as fitting a gaussian to the Binomial distribution with $p = \frac{1}{2}$ . For any particular $m$ , the variance of the Binomial distribution is maximized when $p = \frac{1}{2}$ . Therefore, the Hoeffding inequality is relatively tight when $p = \frac{1}{2}$ . Unfortunately, the Hoeffding approximation is not tight enough for our purposes. In machine learning, our goal is to find a hypothesis with a true error rate far away from $\frac{1}{2}$ where the Hoeffding inequality becomes loose.

There is another bound known as the “realizable bound” which applies only when $k = 0$ . The realizable bound is: $Bin (m, 0, p) = {(1 - p)}^{m} \leq e^{- m p}$ The realizable bound is noticeably tighter with an exponent proportional to $|p - \frac{k}{m}|$ rather than ${(p - \frac{k}{m})}^{2}$ . The disadvantage of the realizable bound is that it only applies to a very limited setting - when our empirical error rate happens to be $0$ .

Luckily, there exists a quickly calculable bound which achieves the generality of the Hoeffding bound along with the tightness of the realizable bound. We have the relative entropy Chernoff bound [7] for $\frac{k}{m} < p$ :

Bin (m, k, p) \leq e^{- m KL (\frac{k}{m} ∣ ∣ p)}

(3.2.1)

Here

KL (q ∣ ∣ p) = q ln \frac{q}{p} + (1 - q) ln \frac{(1 - q)}{(1 - p)}

is the KL-divergence between a coin of bias

q

and another coin of bias

p

. The relative Chernoff bound is as tight as the Hoeffding bound when

p

is near

\frac{1}{2}

and as tight as the realizable bound when

k = 0

. In between the extremes, the relative Chernoff bound smoothly interpolates between these possibilities.

We are concerned with the different bounds here because much of the learning theory literature (see [50], [20], [39] for examples) works with either the realizable bound or the Hoeffding bound, or both. In contrast, we will work with either the relative Chernoff bound or the exact tail probability, $Bin (m, k, p)$ . There are several advantages to this approach:

Sometimes, a different approach to producing a bound will appear better than previous approaches, but the apparent benefit can simply be traced to the use of a tighter bound on $Bin (m, k, p)$ .
The bounds presented here will all be immediately applicable to direct calculation.
We avoid the need to state two versions of the same theorem: once for the realizable ( $0$ empirical error) case and once for the agnostic (arbitrary empirical error) case.

The principle disadvantage of this approach is that both the relative entropy Chernoff bound and $Bin (m, k, p)$ are not analytically invertible. Lack of invertibility is a theoretical disadvantage because it means we can not easily parameterize our “precision” parameter, $ε$ in terms of $δ$ . Nonetheless, this is not a severe computational disadvantage because the quantity $Bin (m, k, p)$ is convex in $p$ implying that a binary search is capable of solving the inequality. The process of (and need for) inversion is discussed next.

[next] [prev] [prev-tail] [front] [up]