|
|
The work in this chapter is joint with Avrim Blum and was first presented at ICML [30] and then in the Machine Learning Journal [31]. The presentation here generalizes, unifies, and improves the earlier work. Microchoice bounds can be thought of as unifying “Self-bounding Learning Algorithms” [17] and the Occam’s razor bound, [5].
The bounds of the previous chapter do not use much structure on the hypothesis space. Yet learning algorithms induce a natural structure. In particular, many learning algorithms work by an iterative process in which they take a sequence of steps each from a small set of choices (small in comparison to the overall hypothesis set size). Local optimization algorithms such as hill-climbing or simulated annealing, for example, work in this manner. Each step in a local optimization algorithm can be viewed as making a choice from a small set of possible steps to take. If we take into account the number of choices made and the size (and other properties) of each choice set, can we produce a tighter bound? This chapter introduce microchoice bounds which use this type of information to construct high confidence bounds on the future error rate.
The microchoice bounds can be thought of in several ways. The simple microchoice bound (given in section 5.2) can be thought of as a well motivated application of the Occam’s Razor bound ( 4.6.1). The idea is to use the learning algorithm itself to define a description language for hypotheses, so that the description length of the hypothesis actually produced gives a bound on the estimation error. The adaptive microchoice theorem (in section 5.3) can be thought of as a computationally tractable adaptation of Self-bounding Learning algorithms [17]. Microchoice bounds tie together and show the relationship between these different sample complexity bounds.
Microchoice bounds also provide insight into the nature of choices. In general, we know that choice is “bad” for the purposes of creating a uniform bound on the true error rate. The microchoice bounds give a quantitative understanding of how much choice is “bad”. In particular, the log of the choice space size is the natural measure of “badness”. This is directly related to the log of the hypothesis space size in the discrete hypothesis bound ( 4.2.1). There is also an indirect relationship with information theory where the log of the alphabet size is an important parameter for specifying the number of bits required to send a message.
Viewed as an interactive proof of learning, microchoice bounds can be described pictorially as in figure 5.0.1.
Important early work developing approximately self-bounding learning algorithms was also done by Domingos [12].
