13.1. Theoretical setup

We first present a modern neural network bound (the “competition”), then specialize the PAC-Bayes bound to a stochastic neural network. A stochastic neural network is simply a neural network where each weight in the neural network is drawn from some distribution whenever it is used. The reason for constructing a stochastic neural network is that it will have a much lower true error upper bound than the neural network. Furthermore, this will be accomplished without increasing the empirical error rate more than marginally.

13.1.1. Neural Network bound

We will compare a specialization of the best current neural network true error rate bound [28] with our approach. The neural network bound is described in terms of the following parameters:

1. A margin, $0<\theta <1$.
2. A function $\phi$ defined by $\phi \left(x\right)=1$ if $x<0$, $\phi \left(x\right)=0$ if $x>1$, and linear in between.
3. $A_i$, an upper bound on the sum of the magnitude of the weights in the $i$th layer of the neural network
4. $L_i$, a Lipschitz constant which holds for the $i$th layer of the neural network. A Lipschitz constant is a bound on the magnitude of the derivative.
5. $d$, the size of the input space.

With these parameters defined, we get the following bound.

The theorem is actually only given up to a universal constant. “$32$” might be the right choice, but this is just an educated guess by the author [42]. The neural network true error bound above is (perhaps) the tightest known bound for general feed-forward neural networks and so it is the natural bound to compare with.

This 2 layer feed-forward bound is not easily applied in a tight manner because we can’t calculate a priori what our weight bound $A_i$ should be. This can be patched up using the principle of structural risk minimization. In particular, we can state the bound for $A_1={\alpha }^j$ where $j$is some non-negative integer and $\alpha >1$ is a constant. If the $j$th bound holds with probability $\frac{\delta }{j\left(j+1\right)}$, then all bounds will hold simultaneously with probability $1-\delta$, since $${\sum }_{j=1}^{\infty }\frac{1}{j\left(j+1\right)}=1$$ Applying this approach to the values of both $A_1$ and $A_2$, we get the following theorem:

In practice, we will use $\alpha =\beta =1.1$ and report the value of the tightest applicable bound for all $j,k$.

13.1.2. Stochastic Neural Network bound

We will specialize a PAC-Bayes bound ( 6.2.1) for application to a stochastic neural network with a choice of the “prior”. Our “prior” will be zero on all neural net structures other than the one we train and a multidimensional isotropic gaussian on the values of the weights in our neural network. The multidimensional gaussian will have a mean of $0$ and a variance in each dimension of $b^2$. This choice is made for convenience and happens to provide good results.

The optimal value of $b$ is unknown and dependent on the learning problem so we will wish to parameterize it in an example dependent manner. We can do this using the same trick as for the original neural net bound. Use a sequence of bounds where $b=c{\alpha }^j$ for $c$ and $\alpha$ some constants and $j$ a nonnegative number. For the $j$th bound set $\delta \to \frac{\delta }{j\left(j+1\right)}$. The union bound will imply that all bounds hold simultaneously with probability at least $1-\delta$.

Assuming that our “posterior” $Q$ is also defined by a multidimensional gaussian with the mean and variance in each dimension defined by $w_i$ and $s_i^2$, we can specialize to the following corollary:

We will choose $\alpha =1.1$ and $c=0.2$ as reasonable default values.

One more step is necessary in order to apply this bound. The essential difficulty is evaluating ${\widehat{e}}_q\left(h\right)$. This quantity is observable although calculating it to high precision is difficult. We will use the Monte Carlo sampling technique of section 6.3.1 in order to bound the value of ${\widehat{e}}_q\left(h\right)$ and then use the bound on this value in the PAC-Bayes bound. We use $n=1000$ evaluations of the empirical error rate of the stochastic neural network.

13.1.3. Distribution Construction algorithm

One critical step is missing in the description: How do we calculate the multidimensional gaussian, $Q$? The variance of the posterior gaussian needs to be dependent on each weight in order to achieve a tight bound since we want any “meaningless” weights to not contribute significantly to the overall sample complexity. We use a simple greedy algorithm to find the appropriate variance in each dimension.

1. Train a neural net on the examples
2. For every weight, $w_i$, search for the variance, $s_i^2$, which reduces the empirical accuracy of the trained network by $1\%$ (for example) while holding all other weights fixed.
3. The stochastic neural network defined by $\left\{w_i,s_i^2\right\}$ will generally have a too-large empirical error. Therefore, we calculate a global multiplier $\lambda <1$ such that the stochastic neural network defined by $\left\{w_i,\lambda s_i^2\right\}$ decreases the empirical accuracy by only $1\%$.
4. Then, we evaluate the empirical error rate of the resulting stochastic neural net with $1000$ samples from the stochastic neural network.