5.1 A Motivating Observation

5.1. A Motivating Observation

Imagine, for the moment, that we know the (unknown) problem distribution, $D$ . For a given learning algorithm $A$ , a distribution $D$ on labeled examples induces a distribution $q (h)$ over the possible hypotheses $h \in H$ produced by algorithm $A$ after $m$ examples¹. A natural choice for the Occam’s Razor bound ( 4.6.1) is the measure $p (h) = q (h)$ . Is this choice optimal? The answer is “yes”, given the right notion of optimal. In particular, if we start with the relative entropy Occam’s razor bound ( 4.6.2), we can show that $p (h) = q (h)$ minimizes the expected value of the bound on the Kullback-Leibler divergence between the empirical error and true error.

THEOREM 5.1.1.

(KL divergence minimization) $p (h) = q (h)$ minimizes the expected value of the KL divergence in the relative entropy Occam’s Razor bound ( 4.6.2).

PROOF. We need to show that $q (h) = {argmin}_{p (h)} \sum_{h} q (h) \frac{ln \frac{1}{p (h)} + ln \frac{1}{δ}}{m}$ removing terms which the minimum does not depend on, we get: ${argmin}_{p (h)} \sum_{h} q (h) ln \frac{1}{p (h)}$ adding a constant, we get: ${argmin}_{p (h)} \sum_{h} q (h) ln \frac{q (h)}{p (h)}$ This is equivalent to minimizing the Kullback-Leibler divergence between the distribution of $q (h)$ and $p (h)$ which is minimized for $q (h) = p (h)$ . ▫

Using the KL divergence as our notion of loss is somewhat non-intuitive. However, it is mathematically simple and not irrational. For all true errors, the KL divergence will be upper and lower bounded between an $l_{1}$ and an $l_{2}$ metric. Since these are two of the most common metrics, the choice of KL divergence based metric should behave similarly well.

The point of these observations is to notice that if the structure of the learning algorithm produces a choice $p (h)$ that approximates $q (h)$ , the result should be better estimation bounds.

[next] [front] [up]