5.2. The Simple Microchoice Bound

The simple microchoice bound is essentially a compelling and easy way to select a measure $p\left(h\right)$ for learning algorithms that operate by making a series of small choices. In particular, consider a learning algorithm that works by making a sequence of choices, $c_1,...,c_d$, from a sequence of choice sets, $C_1,...,C_d$, finally producing a hypothesis, $h\in H$. Specifically, the algorithm first looks at the choice set $C_1$ and the data $z^N$ to produce choice $c_1\in C_1$. The choice $c_1$ then determines the next choice set $C_2$ (different initial choices produce different choice sets for the second level). The algorithm again looks at the data to make some choice $c_2\in C_2$. This choice then determines the next choice set $C_3$, and so on. These choice sets can be thought of as nodes in a choice tree, where each node in the tree corresponds to some internal state of the learning algorithm, and a node containing some choice set $C$ has branching factor $\mid C\mid$. Pictorially, we can draw the tree as follows:

Depending on the learning algorithm, sub-trees of the overall tree may be identical. We address optimization of the bound for this case later. Eventually there is a final choice leading to a leaf, and a single hypothesis is output.

For example, the decision list algorithm of Rivest [45], applied to a set of $n$ features, uses the data to choose one of $4n+2$ rules (e.g., “if ${\bar{x}}_3$ then $-$”) to put at the top. Based on the choice made, it moves to a choice set of $4n-2$ possible rules to put at the next level, then a choice set of size $4n-6$, and so on, until eventually it chooses a rule such as “else $+$” leading to a leaf.

The microchoice bound calculation program is as follows:

1. set $p\leftarrow 1$
2. while learning algorithm has not halted.
   1. $\mid C\mid \leftarrow$ number of possible data-dependent choices
   2. $p\leftarrow \frac{p}{\mid C\mid }$

3. return $p$

Pictorially, this algorithm can be thought of as taking a “supply” of probability at the root of the choice tree.

The root takes its supply and splits it equally among all its children. Recursively, each child then does the same: it takes the supply it is given and splits it evenly among its children, until all of the supplied probability is allocated among the leaves. If we examine some leaf containing a hypothesis $h$, we see that this method gives at least probability $p\left(h\right)={\prod }_{i=1}^{d\left(h\right)}\frac{1}{\mid C_i\left(h\right)\mid }$ to each $h$ for any path of depth $d\left(h\right)$ reaching the hypothesis $h$.

Note it is possible that several leaves will contain the same hypothesis $h$, and in that case one should really add the allocated measures together. However, the microchoice bound neglects this issue, implying that it will be unnecessarily loose for learning algorithms which can arrive at the same hypothesis in multiple ways. The reason for neglecting this is that now, $p\left(h\right)$ is something the learning algorithm itself can calculate by simply keeping track of the sizes of the choice sets it has encountered so far. It is important to notice that this construction is defined before observing any data. Consequently, every hypothesis has some bound associated with it before the data is used to pick a particular hypothesis and its corresponding bound.

Another way to view this process is that we cannot know in advance which choice sequence the algorithm will make. However, a distribution $D$ on labeled examples induces a probability distribution over choice sequences, inducing a probability distribution $q\left(h\right)$ over hypotheses. Ideally we would like to use $p\left(h\right)=q\left(h\right)$ in our bounds as noted above. However, we cannot calculate $q\left(h\right)$ (since the distribution $D$ is unknown), so instead, our choice of $p\left(h\right)$ will be just an estimate. We hope that the algorithm designer has chosen a “good” leaning algorithm which induces a distribution $p\left(h\right)$ over the final hypotheses which is near to $q\left(h\right)$. Our estimate $p\left(h\right)$ is the probability distribution resulting from picking each choice uniformly at random from the current choice set at each level (note: this is different from picking a final hypothesis uniformly at random). I.e., it can be viewed as the measure associated with the assumption that at each step, all choices are equally likely.

We immediately find the following theorem:

Proof. Specialization of the Occam’s Razor bound ( 4.6.1).

Once again, it will be worthwhile to slightly loosen this bound with the following corollary:

The point of the microchoice bound is that the quantity $\bar{e}\left(...\right)$ is something the algorithm can calculate as it goes along, based on the sizes of the choice sets encountered. To see this, note that the hypothesis dependent term is ${\sum }_{i=1}^{d\left(h\right)}ln\mid C_i\left(h\right)\mid$. The quantity $d\left(h\right)$ can be calculated by just noting the number of choices made before the learning algorithm terminates. The choice sets, $C_i\left(h\right)$, can often be easily deduced by reasoning about the possible microchoices the algorithm could have made given different datasets.

In many natural cases, a “fortuitous distribution and target concept” corresponds to a shallow leaf or a part of the tree with low branching, resulting in a better bound. For instance, in the decision list case, ${\sum }_{i=1}^{d\left(h\right)}ln\mid C_i\left(h\right)\mid$ is roughly $dlnn$ where $d$is the length of the list produced and $n$ is the number of features. Notice that $dlnn$ is also the description length of the final hypothesis produced in the natural encoding, thus in this case these theorems yield similar bounds to a simple application of Occam’s razor or SRM.

More generally, the microchoice bound is similar to Occam’s razor or SRM bounds when each $k$-ary choice in the tree corresponds to $logk$ bits in the natural encoding of the final hypothesis $h$. However, sometimes this may not be the case. Consider, for instance, a local optimization algorithm in which there are $n$ parameters and each step adds or subtracts 1 from one of the parameters. Suppose in addition the algorithm knows certain constraints that these parameters must satisfy (perhaps a set of linear inequalities) and the algorithm restricts itself to choices in the legal region. In this case, the branching factor, at most $2n$, might become much smaller if we are “lucky” and head toward a highly constrained portion of the solution space. One could always reverse-engineer an encoding of hypotheses based on the choice tree, but the microchoice approach is much more natural.

There is also an opportunity to use a priori knowledge in the choice of $p\left(h\right)$. In particular, instead of splitting our confidence equally at each node of the tree, we could split it unevenly, according to some heuristic function $g$. If $g$ is “good” it may produce error bounds similar to the bounds when $p\left(h\right)=q\left(h\right)$. In fact, the method of section ( 5.3) where we combine these results with Freund’s query-tree [17] approach can be thought of as an attempt to do exactly this.

5.2.1. Examples

It is difficult to create a bound which is universally better than previous bounds. The microchoice bound can be much better than the discrete hypothesis bound ( 4.2.1) and can be slightly worse. To develop some understanding of how they compare we consider several cases.

5.2.1.1. Greedy Set Cover

Consider a greedy set cover algorithm for learning an OR function over $F$ Boolean features. The algorithm begins with a choice space of size $F+1$ (one per feature or halt) and chooses the feature that covers the most positive examples while covering no negative ones. It then moves to a choice space of size $F$ (one per feature remaining or halt) and chooses the best remaining feature and so on until it halts. If the number of features chosen is $k$ then the microchoice bound is:

$$\epsilon \left(h\right)=\frac{1}{m}\left(ln\frac{1}{\delta }+{\sum }_{i=1}^{k}ln\left(F-i+2\right)\right)\le \frac{1}{m}\left(ln\frac{1}{\delta }+kln\left(F+1\right)\right)$$

The bound of ( 4.2.1) is:

$$\epsilon =\frac{1}{m}\left(ln\frac{1}{\delta }+Fln2\right).$$

If $k$ is small, then the microchoice bound is a lot better, but if $k=O\left(F\right)$ then the microchoice bound is slightly worse than the discrete hypothesis bound. Notice that in this case the microchoice bound is essentially the same as the standard Occam’s razor analysis when one uses $O\left(lnF\right)$ bits per feature to describe the hypothesis.

5.2.1.2. Decision Trees

Decision trees over discrete sets (say, ${\left\{0,1\right\}}^F$) are another natural setting for application of the microchoice bound.

A decision tree differs from a decision list in that the size of the available choice set is larger due to the fact that there are multiple nodes where a new test may be applied. In particular, for a decision tree with $K$ leaves at an average depth of $d$, the choice set size is $K\left(F-d\right)$, giving a bound noticeably worse than the bound for the decision list. This motivates a slightly different decision algorithm which considers only one leaf node at a time. The algorithm adds a new test or decides to never add a new test at this node. In this case, there are $\left(F-d\left(v\right)+1\right)$ choices for a node $v$ at depth $d\left(v\right)$, implying the bound:

5.2.2. Pruning

Decision tree algorithms for real-world learning problems often have some form of “pruning” as in [44] and [41]. The tree is first grown to full size producing a hypothesis with minimum empirical error. Then the tree is “pruned” starting at the leaves and progressing up through the tree toward the root node using some test for the significance of an internal node. An internal node is not significant if the reduction in total error is small in comparison to the complexity of its children. Insignificant internal nodes are replaced with a leaf resulting in a smaller tree.

Microchoice bounds have the property that they incidentally prove a bound for every decision tree which can be found by pruning internal nodes. In particular, one of the choices available when constructing a node is to make the node a leaf. Therefore, if we begin with the tree $T$and then prune to the smaller tree $T^{\prime }$, we can apply the bound ( 5.2.1) to $T^{\prime }$ as if the algorithm had constructed $T^{\prime }$ directly rather than having gone first through the tree $T$. This suggests another possible pruning criterion: prune a node if the pruning would result in an improved microchoice bound. That is, prune if the increase in empirical error is less than the decrease in $\epsilon \left(h\right)$. This pruning criteria is a “pessimistic criteria” [38].

The similarities to SRM are discussed next.

5.2.3. Microchoice and Structural Risk Minimization

The microchoice bound is essentially a compelling application of the Disjoint SRM bound 4.5.1 where the description language for a hypothesis is the sequence of data-dependent choices which the algorithm makes in the process of deciding upon the hypothesis. The hypothesis set $H_i$ is all hypotheses with the same description length in this language.

As an example, consider a binary decision tree with $F$ Boolean features and a Boolean label. The first hypothesis set, $H_1$ will consist of $2$ hypotheses; always false and always true. In general, we will have one hypothesis set for every legal configuration of internal nodes. The size of a hypothesis set where every tree contains $k$ internal nodes will be $2^{k+1}$ because there are $k+1$ leaves each of which can take $2$ values. The weighting $p\left(i\right)$ across the different hypothesis sets is defined by the microchoice allocation of confidence.

The principle disadvantage of the microchoice bound is that the sequence of data-dependent choices may contain redundancy. A different SRM bound with a different set of disjoint hypothesis sets might be able to better avoid redundancy. As an example, assume that we are working with a decision tree on $F$ binary features. There are $F+2$ choices (any of $F$ features or $2$ labels) at the top node. At the next node down there will be $F+1$ choices in both the left and right children. Repeat until a maximal decision tree is constructed. There will be ${\prod }_{i=0}^F{\left(F-i+2\right)}^{2^i}$ possible trees. This number is somewhat larger than the number of Boolean functions on $F$ features: $2^{2^F}$.