6.3. PAC-Bayes Approximations

6.3.1. Approximating the empirical error

In practice, it is not always easy to calculate some of the observable variables in the PAC-Bayes bound. In particular, ${\widehat{e}}_q\left(h\right)$ is not necessarily easy to calculate when $q$ is some continuous distribution. We can avoid the need for a direct evaluation by a Monte Carlo evaluation and a bound on the tail of the Monte Carlo evaluation. Let ${\widehat{e}}_{\widehat{q}}\left(h\right)\equiv {Pr}_{\widehat{q},S}\left(h\left(x\right)\ne y\right)$be the observed rate of failure of $n$ random hypotheses drawn according to $q\left(h\right)$ and applied to a random training example. Once again, we have a familiar Binomial distribution. Direct calculation will give us:

In order to calculate a bound on the expected true error rate, we can first bound the expected empirical error rate ${\widehat{e}}_q\left(h\right)$ with confidence $\frac{\delta }{2}$ then bound the expected true error rate $e_q\left(h\right)$ with confidence $\frac{\delta }{2}$, using our bound on ${\widehat{e}}_q\left(h\right)$. Since the total probability of failure is only $\frac{\delta }{2}+\frac{\delta }{2}=\delta$ our bound will hold with probability $1-\delta$.

6.3.2. Derandomizing the PAC-Bayes bound

It is sometimes desirable to derandomize the PAC-Bayes bound. There are several ways to do this. The next chapter will talk about replacing the randomization over $q\left(h\right)$ with a thresholded average. Another technique is to simply pick a hypothesis according to $q\left(h\right)$. While this would probably be effective in practice, the theoretical guarantees that can be made for this technique are weak. Strong theoretical guarantees can be made for a similar technique.

Suppose we make $n$ draws form $q\left(h\right)$. Let the drawn hypotheses be $\left\{h_1,...,h_n\right\}$. We can form a new distribution $\widehat{q}\left(h\right)$ which is uniform over the $n$ draws. The true error rate of this distribution can be bound with high probability according to the following theorem.

Note the this theorem and the last theorem are essentially the same theorem.

This theorem allows us to do an (incomplete) derandomization. Instead of drawing from $q\left(h\right)$in order to evaluate an input, we can draw from $\widehat{q}\left(h\right)$ which requires a fixed finite number of bits. This may allow for more efficient algorithms, and some people may find it reassuring that every hypothesis in $\left\{h_1,...,h_n\right\}$ has a low empirical error. The same confidence splitting trick of the last section can be used in order to guarantee $e_q\left(h\right)$ is bounded and $e_{\widehat{q}}\left(h\right)$ is bounded given that $e_q\left(h\right)$ is bounded.

It is worth mentioning that no assumption of independence applies to either this theorem or the last theorem since we explicitly control (and create) the independence ourselves. These theorems hold for totally verifiable preconditions.