13.2. Experimental Results

How well can we bound the true error rate of a stochastic neural network? The answer is much better than we can bound the true error rate of a neural network.

Our experimental results take place on a synthetic data set which has 25 input dimensions and one output dimension. Most of these dimensions are useless—simply random numbers drawn from a $N\left(0,1\right)$ Gaussian. One of the 25 input dimensions is dependent on the label. First, the label $y$ is drawn uniformly from $\left\{-1,1\right\}$, then the special dimension is drawn from a $N\left(y,1\right)$ Gaussian. Note that this learning problem can not be solved perfectly because some examples will be drawn from the tail of the gaussian.

The “ideal” neural net to use in solving this problem is a single node perceptron. We will instead use a 2 layer neural net with 2 hidden nodes. This overly large neural net will result in the potential for significant overfitting which makes the bound prediction problem interesting. It is also somewhat more “realistic” if the neural net structure does not exactly fit the learning problem.

All of our data sets will use just $100$ examples. Constructing a non-vacuous bound for a continuous hypothesis space at $100$ examples is quite challenging as indicated by figure 13.2.1. Conventional bounds are hopelessly loose while the stochastic neural network bound is still not as tight as might be desired. There are several notable things about this figure.

1. The SNN upper bound is 2-3 orders of magnitude lower than the NN upper bound.
2. The SNN performs better than expected. In particular, the SNN true error rate is closer than $1\%$ of the NN true error rate. This is surprising considering that we fixed the difference in empirical error rates at $1\%$.
3. The SNN bound has a minimum at 12000 pattern presentations which weakly predicts the overfitting point of 6000 for both the SNN and the NN.

The comparison between the neural network bound and the stochastic neural network bound is not quite “fair” due to the form of the bound. In particular, the stochastic neural network bound can never return a value greater than “always err”. This implies that when the bound is near the value of “$1$”, it is difficult to judge how rapidly extra examples will improve the stochastic neural network bound. We can judge the sample complexity of the stochastic bound by plotting the value of the numerator in equation 13.1.1. Figure 13.2.2 plots the complexity versus the number of pattern presentations in training.

The stochastic bound is a radical improvement on the neural network bound but it is not yet a perfectly tight bound. Given that we do not have a perfectly tight bound, one important consideration arises: does the minimum of the stochastic bound predict the minimum of the true error rate (as predicted by a large holdout data set). In particular, can we use the stochastic bound to determine when we should cease training? The stochastic bound depends upon (1) the complexity which increases with training time and (2) the training error which decreases with training time. This dependence results in a minima which for our problem occurs at approximately 12000 pattern presentations. The point of minimal true error (for the stochastic and deterministic neural networks) occurs at approximately 6000 pattern presentations indicating that the stochastic bound weakly predicts the point of minimum error. The neural network bound has no such minimum.

Is the choice of 5% increased empirical error optimal? In general, the “optimal” choice of the extra error rate depends upon the learning problem. Since the stochastic neural network bound (corollary 13.1.3) holds for all multidimensional gaussian distributions, we are free to optimize the choice of distribution in anyway we desire. Figure 13.2.2 shows the resulting bound for different choices of $q$. The bound has a minimum at $0.03$ extra error indicating that a slightly lower bound is possible if we accept a larger training error. Also note that the complexity always decreases with increasing entropy in the distribution of our stochastic neural net. The existence of a minimum in Figure 13.2.2 is the “right” behavior: the increased empirical error rate is significant in the calculation of the true error bound.