This document is primarily about the theory of sample complexity for answering the question “Have we learned?”. However, we do not neglect the experimental side. In particular, following the theory we will present results for application of sample complexity bounds to machine learning problems. These results are the ’best known results’ in terms of bound tightness and should be considered as a guide and challenge to others working on sample complexity bounds.
All of the sample complexity bounds presented here will fall within the paradigm of classical (non-Bayesian) statistics. Despite this, Bayesians may be interested in the results. In particular, it is worth noting that we will consider the use of a ’prior’ and a ’posterior’ (in a classical manner) within these bounds.
In order to make this thesis more coherent, previous work (of which there is quite a bit) will be integrated into the presentation rather than separated into a section of its own. Credit will be given at the time the work is introduced.
The document is organized into 3 parts:
What follows is a brief chapter-by-chapter summary of the theoretical results in this thesis. Much of the work can be summarized as approaches which use extra information (in the algorithm, in error rates of hypotheses which are not chosen, in test sets, etc...) in order to construct tighter bounds on the future error rate of a hypothesis.
The principal practical result of this thesis is the construction of a program ’bound’ which automatically uses any of several theorems in calculating a true error bound on the future error rate of a particular hypothesis. The use of this program will be demonstrated as the bounds are presented.
The Microchoice technique allows for online construction of a “prior” which can be used in the Occam’s Razor bound ( 4.6.1). Empirical testing (presented in chapter 11) shows that this approach is practical and yields useful results on decision trees for real-world problems drawn from the UCI database of problems. The Adaptive Microchoice bounds further extend this approach and can result in functional improvements over the Occam’s Razor bound.
Pac-Bayes bounds are a new approach (first presented by David McAllester [39]) for for dealing with continuously parameterized classifiers such as (stochastic) neural networks. This chapter refines and improves the PAC-Bayes bound, giving it an information theoretic interpretation. Empirical results (presented in chapter 12) show that refined PAC-Bayes bound works to produce nonvacuous bounds on realistic learning problems. Nonvacuous bounds for continuous-valued classifiers are currently rare.
Averaging bounds deal with classifiers formed by picking according to the weighted majority on some other set of classifiers. Averaging is a very common technique in practice (see [?] for examples), so results specialized for averaging classifiers are useful. This chapters states and proves a bound on averaging classifiers which shows that “hypothesis space complexity” decreases as averaging becomes more uniform. Prior theoretical work [?] was of the form “averaging does not increase the hypothesis space complexity much”.
Shell bounds are a new approach which trades extra information for tighter bounds. Empirical results in (presented in chapter 11) show this can be a useful approach. In order to ameliorate the increased information requirements, a sampled version of the bound is stated which allows for smooth interpolation between simpler bounds and the full shell bound. In addition, the shell bound has been extended to continuous spaces with an approach similar to PAC-Bayes bounds.
The results of this chapter are entirely theoretical. They point out an approach which may lead to useful technique for bounds on continuous valued hypothesis spaces. Using a stronger notion of cover (the bracketing cover), simplified bounds on the future error rate of continuous classifiers can be constructed. This approach can be significantly tighter than the standard covering number approach. More work in calculation of bracketing covers is needed before these results can be applied.
Progressive Validation is a variant of the holdout bound ( 4.1.1) which is roughly twice as efficient about how it uses examples. Progressive Validation is not used in the experimental results chapter(s) because more work is required in order to prove the bound without a Hoeffding-like approximation.
The idea behind this chapter is that it should be possible to combine any test set bound with any training set based bound in order to derive a bound with more robust behavior. A general technique is stated and proved to work. Empirical results (in chapter 11) show that this approach can work well in practice.