6.2 A Tighter PAC-Bayes Bound

We can tighten this bound by employing a more accurate tail bound on the Binomial distribution. The proof of this improved lemma is not as straightforward as a simple substitution of the Hoeffding bound with the relative entropy Chernoff bound 3.2.1 but it can be worked out nonetheless.

This bound is always at least as tight as the original PAC-Bayes bound [39] and sometimes much tighter, such as when the average empirical error is near

0

. In particular, when the average empirical error is zero (

{\hat{e}}_{q} (h) = 0

) the bound can be significantly tighter as shown in figure 3.4.1 on page 56.

One interesting new feature of this PAC-Bayes bound is “dimensionally consistency”¹. In particular, each side of the equation is an expectation of log probabilities—“nats”. Rewriting, we get that with high probability, approximately the following holds:

m KL ({\hat{e}}_{q} (h) ∣ ∣ e_{q} (h)) \leq KL (q ∣ ∣ p)

There is a coding theory interpretation of KL divergence:

KL (q ∣ ∣ p) =

the expected number of extra bits required to encode symbols drawn from

q

given a code designed for symbols drawn from

p

rather than from

q

Using the coding theory interpretation of KL divergence, this says approximately: “With high probability the number of extra bits required to encode the empirical errors is less than the number of extra bits required to encode hypotheses drawn from the posterior.”

The retrofit of the PAC-Bayes bound is accomplished by reproving a technical lemma about distributions. The proof relies upon two lemmas. The first is Lemma 22 from [39] which is given by:

PROOF. given in [39]. ▫

We will need to prove the following lemma in order to tighten the PAC-Bayes bound. It is analogous to Lemma 17 from [39].

PROOF. For any given hypothesis $h$ we will prove the following.

\forall h E_{D^{m}} e^{(1 - α) m KL (\hat{e} (h) ∣ ∣ e (h))} \leq \frac{1}{α}

(6.2.1)

The Lemma then follows from the sequence:

\Rightarrow \forall p E_{p} E_{D^{m}} e^{(1 - α) m KL (\hat{e} (h) ∣ ∣ e (h))} \leq \frac{1}{α}

$\Rightarrow \forall p E_{D^{m}} E_{p} e^{(1 - α) m KL (\hat{e} (h) ∣ ∣ e (h))} \leq \frac{1}{α}$ $\Rightarrow \forall p {Pr}_{D^{m}} (E_{p} e^{(1 - α) m KL (\hat{e} (h) ∣ ∣ e (h))} \geq \frac{1}{α δ}) \leq δ$

Consequently, we must only prove equation 6.2.1. Given the hypothesis, we have a fixed true error rate, $e (h)$ , and the empirical error rate $\hat{e} (h)$ will be distributed like a Binomial. Let $R (e (h))$ be the random variable with a cumulative distribution given by the relative entropy Chernoff bound for a hypothesis with true error $e (h)$ . In other words, define a cumulative distribution function on $[0, 1]$ according to: $e^{- m KL (R ∣ ∣ p)}$ (note that we defined $KL ()$ so that it is always $0$ when $\frac{k}{m} > p$ ). Note that the relative entropy Chernoff bound implies $R$ satisfies: $Bin (m, m R, p) \leq e^{- m KL (R ∣ ∣ p)}$ whenever $m R$ is integer.

Since $e^{(1 - α) m KL (\hat{e} (h) ∣ ∣ e (h))}$ increases monotonically with decreasing $\hat{e} (h)$ , the probability distribution function of $R (e (h))$ will have a larger expected value. In other words: $E_{D^{m}} e^{(1 - α) m KL (\hat{e} (h) ∣ ∣ e (h))} \leq E_{R} e^{(1 - α) m KL (R ∣ ∣ e (h))}$ The probability distribution function of $R$ is given by: $f (x) = \{\begin{matrix} 0 & for x \geq p \\ - m \frac{\partial KL (x ∣ ∣ p)}{\partial x} e^{- m KL (x ∣ ∣ p)} & for x \leq p \end{matrix}$ Taking the expectation with respect to this distribution gives us:

\begin{array}{rcl} E_{R} [e^{(1 - α) m KL (R ∣ ∣ e (h))}] & = & \int_{0}^{1} e^{(1 - α) m KL (x ∣ ∣ e (h))} f (x) d x \\ = & \int_{0}^{e (h)} - m \frac{\partial KL (x ∣ ∣ e (h))}{\partial x} e^{- α m KL (x ∣ ∣ e (h))} d x \\ = & {\frac{1}{α} e^{- α m KL (x ∣ ∣ e (h))}|}_{0}^{e (h)} \\ \leq & \frac{1}{α} \end{array}

This finishes the proof of the technical lemma. ▫

PROOF. First, we can specialize lemma 6.2.3 with $α = \frac{1}{m}$ to get that with probability $1 - δ$ $E_{p} e^{(m - 1) KL (\hat{e} (h) ∣ ∣ e (h))} \leq \frac{m}{δ}$ Apply lemma 6.2.2 with $K = \frac{m}{δ}$ , $β = m - 1$ , $Q_{i} = q (h_{i})$ and $y_{i} = KL (\hat{e} (h) ∣ ∣ e (h))$ to get: $E_{q} KL (\hat{e} (h) ∣ ∣ e (h)) = \sum_{i = 1}^{n} Q_{i} KL (\hat{e} (h_{i}) ∣ ∣ e (h_{i})) \leq \frac{KL (q ∣ ∣ p) + ln \frac{m}{δ}}{m - 1}$ Jensen’s inequality, gives us: $KL ({\hat{e}}_{q} (h) ∣ ∣ e_{q} (h)) \leq E_{q} KL (\hat{e} (h) ∣ ∣ e (h)) \leq \frac{KL (q ∣ ∣ p) + ln \frac{m}{δ}}{m - 1}$ which proves the theorem for the finite case. For the infinite case, a sequence of limits can be defined just as in [39]. ▫

6.2. A Tighter PAC-Bayes Bound