4.6 Incorporating a “Prior”

4.6. Incorporating a “Prior”

In constructing the discrete hypothesis bound ( 4.2.1), it is notable that an arbitrary choice was made. We decided to give the same error allowance to every hypothesis. This is an arbitrary choice which, in practice, we will wish to make differently. The next theorem is essentially a restatement of the discrete hypothesis bound with this arbitrary choice made explicit. The same bound using the Hoeffding approximation has appeared elsewhere [5][39].

THEOREM 4.6.1.

(Occam’s Razor Bound) For all hypothesis spaces, $H$ , for all “priors” $p (h)$ over the hypothesis space, $H$ , for all $δ \in (0, 1]$ : $\forall p (h) {Pr}_{D^{m}} (\exists h \in H : e (h) \geq \bar{e} (m, \hat{e} (h), δ p (h))) \leq δ$ where $\bar{e} (m, \frac{k}{m}, δ) \equiv {max}_{p} {p : Bin (m, k, p) = δ}$

It is very important to notice that the “prior” $p (h)$ must be selected before seeing the examples.

PROOF. The proof again starts with the basic observation that: ${Pr}_{D^{m}} (e (h) \geq \bar{e} (m, \hat{e} (h), δ)) \leq δ$ then, we apply the union bound in a nonuniform manner. In particular, we allocate confidence $δ p (h)$ to hypothesis $h$ . Since $p$ is normalized, we know that $\sum_{h} δ p (h) = δ$ which implies that the union bound completes the proof. ▫

Once again, we can relax the Occam’s Razor bound (theorem 4.6.1) with the relative entropy Chernoff bound ( 3.2.1) to get a somewhat more tractable expression.

COROLLARY 4.6.2.

(Relative Entropy Occam’s Razor Bound) For all hypothesis spaces, $H$ , for all “priors” $p (h)$ over the hypothesis space, $H$ , for all $δ \in (0, 1]$ : ${Pr}_{D^{m}} (\exists h \in H : KL (\hat{e} (h) ∣ ∣ e (h)) \geq \frac{ln \frac{1}{p (h)} + ln \frac{1}{δ}}{m}) \leq δ$

PROOF. approximate the Binomial tail with ( 3.2.1) and solve for the minimum. ▫

The Occam’s razor bound is often nonvacuous for discrete learning algorithms such as decision lists and decision trees. The next chapter will discuss a particular motivated choice of the measure $p (h)$ which can lead to much tighter bounds in practice.

The application of the Occam’s Razor bound is somewhat more complicated then the application of the test set bound. Pictorially, the protocol for bound application is given in figure 4.6.1.

Figure 4.6.1:

In order to apply this bound it is necessary that the choice of “Prior” be made before seeing any training examples. Then, the bound is calculated based upon the chosen hypothesis. Note that it is “legal” to chose the hypothesis based upon the prior

p (h)

as well as the empirical error

{\hat{e}}_{S} (h)

Examples of calculation of these bounds is detailed in appendix section 16.2.

The “Occam’s Razor bound” is strongly related to compression. In particular, for any self-terminating description language, $d (h)$ , we can associate a “prior” $p (h) = 2^{- d (h)}$ with the property that $\sum_{h} p (h) \leq 1$ . Consequently, short description length hypotheses will tend to have a tighter convergence and the penalty term, $ln \frac{1}{p (h)}$ is the number of “nats” (bits base e).

[prev] [prev-tail] [front] [up]