6.1 PAC-Bayes Basics

6.1. PAC-Bayes Basics

In the PAC-Bayes setting, a classifier is defined by a distribution $q (h)$ over the hypothesis space. Each classification is carried out according to a hypothesis sampled from $q (h)$ . We are interested in the gap between the expected generalization error $e_{q} (h) \equiv E_{q} [e (h)]$ and the expected empirical error ${\hat{e}}_{q} (h) = E_{q} [\hat{e} (h)]$ , where both expectations are taken with respect to $q (h)$ . The gap will be parameterized by the Kullback-Leibler divergence (see [10]). Recall that:

KL (q ∣ ∣ p) = E_{h \sim q (h)} ln \frac{q (h)}{p (h)}

(6.1.1)

If the support is finite, we have

KL (q ∣ ∣ p) = \sum_{h} q (h) ln \frac{q (h)}{p (h)}

(6.1.2)

The relative entropy is an asymmetric distance measure between probability distributions, with

KL (q ∣ ∣ p) = 0 \Leftrightarrow q = p

almost everywhere.

THEOREM 6.1.1.

(PAC-Bayes [39]) For all “priors” $p (h)$ and for all $δ \in (0, 1]$ :

${Pr}_{D^{m}} (\exists q (h) : e_{q} (h) \geq {\hat{e}}_{q} (h) + \sqrt{\frac{KL (q ∣ ∣ p) + ln \frac{m}{δ} + 2}{2 m - 1}}) \leq δ$

PROOF. Given in [39]. ▫

This PAC-Bayes bound is almost the same as the Occam’s Razor bound (theorem 4.6.1) when the distribution is peaked on a single hypothesis and the Occam’s razor bound is proved using the looser Hoeffding inequality. This can be seen by noting that the KL-divergence when $q$ is all on one hypothesis, $h$ satisfies $KL (q ∣ ∣ p) = log \frac{1}{p (h)}$ . Comparing with the Occam’s Razor bound, we see that a (small) extra term of size $\frac{ln m}{m}$ has been introduced in return for the capability to average with respect to any posterior $q (h)$ . It is unclear yet that this $\frac{ln m}{m}$ term needs to be there.

PROBLEM 6.1.2.

(Open) Remove the $\frac{ln m}{m}$ term from the sample complexity.

The real power of the PAC-Bayes bound occurs when the average is over many hypotheses. This might occur if the distribution $q (h)$ is picked using Bayes law or a Gibbs distribution. One of the most interesting aspects of the PAC-Bayes bound is that it holds for finite and infinite hypothesis spaces.

[next] [front] [up]