PAC-Bayes bounds give excellent results on a stochastic neural network. The stochastic neural network bound is radically tighter ( orders of magnitude) bound on the true error rate of a classifier while increasing the empirical and true error rates only a small amount.
Although, the stochastic neural net bound is not completely tight, it is not vacuous with just examples and the minima of the bound weakly predicts the point where overtraining occurs.
(Open) To what extent do these results extend to other learning problems and other continuous learning algorithms? Work is under-way (most notably in [48]) to evaluate both of these questions.