7.5. Final thoughts for Averaging Bounds

The practical implication of this more-refined analysis of averaging bounds is more support for the practice of averaging. Sufficient averaging over the hypothesis space can reduce the “complexity” allowing tight estimates on the true error rate—possibly even tighter than could be guaranteed with a single hypothesis.

The improved averaging bound is not yet the tightest possible and it appears there are several possible theoretical improvements.

1. Remove the low order terms from the bound to make it more quantitatively applicable.
2. Improve the argument to take into account the distribution of the margin rather than the margin at some point.
3. Prove a lower bound which corresponds to the upper bound given here. Since no good lower bound yet exists, we do not know that large improvements in the upper bound are not possible.

Open Problem: In practice, is the averaging bound ( 7.3.15) ever better than the PAC-Bayes bound 6.2.1? The extra triangle inequality applications in the averaging bound may make it worse than the PAC-Bayes bound in practice.