5.3. Combining Microchoice with Freund’s Query Tree approach

The next section is devoted to an improvement of the microchoice bound called adaptive microchoice, which arises from synthesizing Freund’s query trees [17] with the microchoice bound. This improvement is not easily expressed as a simplification of Structural Risk Minimization. In essence, the adaptive microchoice bound can gain from dependence on the learning problem distribution $D$ and can take advantage of an “easy” distribution.

First we require some background material in order to state and understand Freund’s bound.

5.3.1. Preliminaries and Definitions

The statistical query framework introduced by Kearns [26] restricts learning algorithm to only access the data using statistical queries. A statistical query takes as input a binary predicate, $\chi$, mapping examples to a binary output: $\left(X,Y\right)\to \left\{0,1\right\}$. The output of the statistical query is the average of $\chi$ over the examples seen. Let $z_i$ be the $i$th labeled example, then: $${\widehat{D}}_{\chi }=\frac{1}{m}{\sum }_{i=1}^m\chi \left(z_i\right)$$

The output is an empirical estimate of the true value $D_{\chi }={\text{E}}_D\left[\chi \left(z\right)\right]={Pr}_{z\sim D}\left(\chi \left(z\right)=1\right)$of the query under the distribution $D$² . One simple example of a predicate $\chi$ is “the first bit is $1$”. A more complicated predicate might be “the third bit xor the 4th bit is $0$”. Naturally, the distribution of ${\widehat{D}}_{\chi }$ will be the familiar Binomial distribution.

It is convenient to define $${\bar{I}}_{D_{\chi }}\left(\delta \right)=\frac{1}{m}{max}_k\left\{k:1-\text{Bin}\left(m,k,D_{\chi }\right)\ge \frac{\delta }{2}\right\}$$ and $${\underset{‾}{I}}_{D_{\chi }}\left(\delta \right)=\frac{1}{m}{min}_k\left\{k:\text{Bin}\left(m,k,D_{\chi }\right)\ge \frac{\delta }{2}\right\}$$ and let $$I_{\chi }\left(\delta \right)=\left[{\underset{‾}{I}}_{D_{\chi }}\left(\delta \right),{\bar{I}}_{D_{\chi }}\left(\delta \right)\right]$$ Intuitively, $I_{\chi }\left(\delta \right)$ is a (fixed) interval in which the random variable ${\widehat{D}}_{\chi }$ will fall with high probability. In other words, we know that: $$Pr\left[{\widehat{D}}_{\chi }/\in I_{\chi }\left(\delta \right)\right]\le \delta$$ Now, we want to construct a confidence interval based upon the high confidence interval $I_{\chi }\left(\delta \right)$. We can do this using the inversion lemma ( 3.4.1) to get: $${\bar{D}}_{\chi }\left(\delta \right)={max}_p\left\{p:{\underset{‾}{I}}_p\left(\delta \right)={\widehat{D}}_{\chi }\right\}$$ and $${\underset{‾}{D}}_{\chi }\left(\delta \right)={min}_p\left\{p:{\bar{I}}_p\left(\delta \right)={\widehat{D}}_{\chi }\right\}$$

The random interval defined here contains the “real” answer $D_{\chi }$ with high probability. In other words, we have: $$Pr\left[D_{\chi }/\in \left[{\underset{‾}{D}}_{\chi }\left(\delta \right),{\bar{D}}_{\chi }\left(\delta \right)\right]\right]\le \delta$$

5.3.2. Background and Summary

Freund [17] considers choice algorithms that at each step perform a Statistical Query on the sample, using the result to determine which choice to take. For an algorithm $A$, tolerance $\alpha$ (defined next), and distribution $D$, Freund defines the query tree $T_A\left(D,\alpha \right)$ as the choice tree created by considering only those choices resulting from answers ${\widehat{D}}_{\chi }$ to queries $\chi$ such that $\left|{\widehat{D}}_{\chi }-D_{\chi }\right|\le \alpha$. The idea is that if a particular predicate, $\chi$, is true with probability $.9$ (for example) on a random sample it is very unlikely that the empirical result of the query will be $.1$. More generally, the chance the answer to a given query is off by more than $\alpha$ is at most $2e^{-2m{\alpha }^2}$ by Hoeffding’s inequality. So, if the entire tree contains a total of $\mid Q\left(T_A\left(D,\alpha \right)\right)\mid$ queries in it, the probability any of these queries is off by more than $\alpha$ is at most $2\cdot \mid Q\left(T_A\left(D,\alpha \right)\right)\mid \cdot e^{-2m{\alpha }^2}$. In other words, this is an upper bound on the probability the algorithm ever “falls off the tree” and makes a low probability choice. The point of this is that we can allocate half (say) of the confidence parameter $\delta$ to the event that the algorithm ever falls off the tree, and then spread the remaining half evenly on the hypotheses in the tree (which hopefully is a much smaller set than the entire hypothesis set).

Unfortunately, the query tree suffers from the same problem as the $q\left(h\right)$ distribution considered in section ( 5.1), namely that to compute it, one needs to know $D$. So, Freund proposes an algorithmic method to find a super-set approximation of the tree. The idea is that by analyzing the results of queries, it is possible to determine which outcomes were unlikely given that the query is close to the desired outcome. In particular, each time a query $\chi$ is asked and a response ${\widehat{D}}_{\chi }$ is received, if it is true that $\mid {\widehat{D}}_{\chi }-D_{\chi }\mid \le \alpha$, then the range $\left[{\widehat{D}}_{\chi }-2\alpha ,{\widehat{D}}_{\chi }+2\alpha \right]$ contains the range $\left[D_{\chi }-\alpha ,D_{\chi }+\alpha \right]$. Thus, under the assumption that no query in the correct tree is answered badly, a super-set of the correct tree can be produced by exploring all choices resulting from responses within $2\alpha$ of the response actually received. By applying this method to every node in the query tree we can generate an empirically observable super-set of the query tree: that is, the original query tree is a pruning of the empirically constructed tree.

A drawback of this method is that it can easily take exponential time to produce the approximate tree, because even the smaller correct tree can have a size exponential in the running time of the learning algorithm. Instead, we would much rather simply keep track of the choices actually made and the sizes of the nodes actually followed, which is what the microchoice approach allows us to do. As a secondary point, given $\delta$, computing a good value of $\alpha$ for Freund’s approach is not trivial, see [17]; we will be able to finesse that issue and use the tighter bound ${\widehat{D}}_{\chi }\in I_{\chi }\left(\delta \right)$.

In order to apply the microchoice approach, we modify Freund’s query tree so that different nodes in the tree receive different confidence, $\delta$, much in the same way that different hypotheses $h$ in our choice tree receive different values of $\delta \left(h\right)$.

5.3.3. Microchoice Bounds for Query Trees

The manipulations of the choice tree are now reasonably straightforward. We begin by describing the true microchoice query tree and then give the algorithmic approximation. As with the choice tree in section ( 5.2), one should think of each node in the tree as representing the current internal state of the algorithm.

We incorporate Freund’s approach into the choice tree construction by having each internal node allocate a portion, $p^{\prime }$ of its “supply” of failure probability to the event that ${\widehat{D}}_{\chi }/\in I_{\chi }\left(\delta \ast p^{\prime }\right)$. The node then splits the remainder of its supply evenly among the children corresponding to choices that result from answers ${\widehat{D}}_{\chi }$ with ${\widehat{D}}_{\chi }\in I_{\chi }\left(\delta \ast p^{\prime }\right)$. Choices that would result from “bad” answers to the query are pruned away from the tree and get nothing. This continues down the tree to the leaves. Pictorially, this looks like:

How should $p^{\prime }$ be chosen? Smaller values of $p^{\prime }$ result in larger intervals $I_{\chi }\left(\delta \ast p^{\prime }\right)$ leading to more children in the pruned tree and less confidence given to each. Larger values of $p^{\prime }$ result in less left over to divide among the children. Unfortunately, our algorithmic approximation (which only sees the empirical answers ${\widehat{D}}_{\chi }$ and needs to be efficient) will not be able to make this optimization. Therefore, we define $p^{\prime }$ in the true microchoice query tree to be $\frac{p}{d+1}$ where $d$ is the depth of the current node. This choice will imply that the adaptive microchoice bound is never much worse than the Microchoice bound, and sometimes much better.

Since a particular query value ${\widehat{D}}_{\chi }$ implies a particular choice $c$, we can think of the interval $I_{\chi }\left(\delta \right)$ as containing choices rather than query results. After all, we only care about the choices the algorithm makes. We can calculate the probability assigned to a hypothesis in the true adaptive microchoice query tree according to the following algorithm:

1. set $p\leftarrow 1$
2. set $d=1$
3. while learning algorithm has not halted.
   1. $d\leftarrow d+1$
   2. $p^{\prime }\leftarrow \frac{p}{d}$
   3. Let $C$ = the current set of possible data-dependent choices.
   4. $\mid \widehat{C}\mid \leftarrow \mid \left\{c\in C\mid c\in I_{\chi }\left(\delta \ast p^{\prime }\right)\right\}\mid$
   5. $p\leftarrow \frac{p-p^{\prime }}{\mid \widehat{C}\mid }$

4. return $p$

There are two important things to note about this algorithm. First of all, we could plug the value $p\left(h\right)$ it returns into the Occam’s Razor Bound 4.6.1 and receive a bound on the true error rate of our chosen classifier.

Second, this algorithm can not be executed. The essential problem is determining whether or not $c\in I_{\chi }\left(\delta \ast p^{\prime }\right)$ which cannot be done without knowledge of the underlying distribution $D$. However, we can calculate an approximate version of this algorithm which, with high probability, returns a value $p$ which is smaller. Since a smaller value is pessimistic, we can use it in our bounds.

The algorithmic approximation uses the idea in [17] of including all choices within the double confidence interval of the observed value ${\widehat{D}}_{\chi }$. Unlike [17], however, we do not actually create the tree; instead we just follow the path taken by the learning algorithm, and argue that the “supply” probability remaining at the leaf is no greater than the amount that would have been there in the original tree. Finally, the algorithm outputs a bound calculated with $p\left(h\right)$.

Specifically, the algorithm is as follows. Suppose we are at a node of the tree containing statistical query $\chi$ at depth $d\left(\chi \right)$ and we have a $p$ supply of parameter. (If the current node is the root, then $p=1$ and $d\left(\chi \right)=1$). We choose $p^{\prime }=p/\left(d+1\right)$, ask the query $\chi$, and receive ${\widehat{D}}_{\chi }$. Let $${\overline{\stackrel{̄}{D}}}_{\chi }\left(\delta \right)={min}_k\left\{\frac{k}{m}:1-\text{Bin}\left(m,k,{\bar{D}}_{\chi }\left(\delta \right)\right)\le \frac{\delta }{2}\right\}$$ and $${\underset{‾}{\underset{‾}{D}}}_{\chi }\left(\delta \right)={max}_k\left\{\frac{k}{m}:\text{Bin}\left(m,k,{\bar{D}}_{\chi }\left(\delta \right)\right)\le \frac{\delta }{2}\right\}$$ with $${\widehat{I}}_{\chi }\left(\delta \right)=\left[{\underset{‾}{\underset{‾}{D}}}_{\chi }\left(\delta \right),{\overline{\stackrel{̄}{D}}}_{\chi }\left(\delta \right)\right]$$

We now let $k$ be the number of children of our node corresponding to answers in the range ${\widehat{I}}_{\chi }\left(p^{\prime }\right)$. We then go to the child corresponding to the answer ${\widehat{D}}_{\chi }$ that we received, giving this child a confidence parameter supply of $\left(p-p^{\prime }\right)/k$. This is the same as we would have given it had we allocated $p-p^{\prime }$ to the children equally. We then continue from that child. Finally, when we reach a leaf, we output the probability left for the hypothesis. Pictorially, this looks like:

Notice that the second choice set is larger than in the true adaptive microchoice set tree. This can easily happen and it makes our results somewhat more pessimistic. The approximate adaptive microchoice algorithm is specified as follows:

1. set $p\leftarrow 1$
2. set $d=1$
3. while learning algorithm has not halted.
   1. $d\leftarrow d+1$
   2. $p^{\prime }\leftarrow \frac{p}{d}$
   3. Let $C$ = the current set of possible data-dependent choices.
   4. $\mid \widehat{C}\mid \leftarrow \mid \left\{c\in C\mid c\in {\widehat{I}}_{\chi }\left(\delta \ast p^{\prime }\right)\right\}\mid$
   5. $p\leftarrow \frac{p-p^{\prime }}{\mid \widehat{C}\mid }$

4. return $p$

Let $d\left(h\right)$ be the depth of some hypothesis $h$ in the empirical path and ${\widehat{C}}_1\left(h\right)$, ${\widehat{C}}_2\left(h\right)$, $\dots$, ${\widehat{C}}_d\left(h\right)$ be the sequence of choice sets resulting in $h$ in the algorithmic construction; i.e., ${\widehat{C}}_i\left(h\right)$ is the number of unpruned children of the $i$-th node. Then, the confidence placed on $h$ will be:

The corresponding relative entropy corollary is:

The bound in theorem ( 5.3.3) is very similar to ( 5.2.2) except that the choice complexity is slightly worsened with the $ln\left(d\left(h\right)+1\right)$ term but improved by replacing $C_i\left(h\right)$ with the smaller ${\widehat{C}}_i\left(h\right)$.

5.3.4. Allowing batch queries

Most natural Statistical Query algorithms make each choice based on responses to a set of queries, not just one. For instance, to decide what variable to put at the top of a decision tree, we ask $F$ queries, one for each feature; we then choose the feature whose answer was most “interesting”. This suggests generalizing the query tree model to allow each tree node to contain a set of queries, executed in batch. Requiring each node in the query tree to contain just a single query as in the above construction would result in an unfortunately high branching factor just for the purpose of “remembering” the answers received so far. ³

Extending the algorithmic construction to allow for batch queries is easily done. If a node has $q$ queries ${\chi }_1,\dots ,{\chi }_q$, we choose the query confidence $\delta \ast p^{\prime }$ as before, but we now split the mass evenly among all $q$ queries. We then let $k$ be the number of children corresponding to answers to the queries ${\chi }_1,\dots ,{\chi }_q$ in the ranges ${\widehat{I}}_{{\chi }_1}\left(\delta p^{\prime }/q\right),\dots ,{\widehat{I}}_{{\chi }_q}\left(\delta p^{\prime }/q\right)$ respectively. We then go to the child corresponding to the answers we actually received, and as before give the child a probability supply of $\left(p-p^{\prime }\right)/k$. Theorem ( 5.3.3) holds exactly as before; the only change is that $\mid {\widehat{C}}_i\left(h\right)\mid$ means the size of the $i$-th choice set in the batch tree rather than the size in the single-query-per-node tree.

5.3.5. Example: Batch Queries for Decision trees

When growing a decision tree, it is natural to make a batch of queries and then make a decision about which feature to place in a node. The process is then repeated to grow the full tree structure. As in the decision tree example described in the simple microchoice section, if we have $F$ features and are considering adding a node at depth $d\left(v\right)$, there are $F-d\left(v\right)+1$ possible features that could be chosen for placement in a particular node. The decision of which feature to use is made by comparing the results of $F-d\left(v\right)+1$ queries to pick the best feature according to some criteria, such as information gain. We can choose $p^{\prime }=p/\left(d+1\right)$, then further divide $p^{\prime }$ into confidences of size $p^{\prime }/\left(F-d\left(v\right)+1\right)$, placing each divided confidence on one of the $F-d\left(v\right)+1$ statistical queries. We now may be able to eliminate some of the $F-d\left(v\right)+1$ choices from consideration, allowing the remaining confidence, $p-p^{\prime }$ to be apportioned evenly amongst the remaining choices. Depending on the underlying distribution this could substantially reduce the size of the choice set. The best case occurs when one feature partitions all examples reaching the node perfectly and all other features are independent of the target. In this case the choice set will have size $1$ if there are enough examples.

5.3.6. Adaptive Microchoice vs. Basic Microchoice

The adaptive microchoice bound is a significant improvement over the simple microchoice bound when the distribution is such that each choice is clear. For example, consider $F$ Boolean features and $N=O\left(F\right)$ examples. Suppose that one feature is identical to the label and all the rest of the features are determined with a coin flip independent of the label.

When we apply a decision tree to a data set generated with this distribution, what will be the resulting bound? Given enough examples, with high probability there will only be one significant choice for the first batch query: the feature identical to the label. The second and third batch queries, corresponding to the children of the root feature, will also have a choice space of size $1$ with very high probability. The “right” choice will be the label value. Each choice set has size $1$ resulting in a complexity of $ln4$ due to allocation of confidence to the statistical queries necessary for learning the decision tree. $ln4$ is considerably better than $ln\left(F+2\right)+2ln\left(F+1\right)$ which the simple version of the microchoice bound provides. Note that the complexity reduction only occurs with a large enough number of examples $m$ implying that the value of $\bar{e}$ calculated can improve faster than (inverse) linearly in the number of examples.

The adaptive microchoice bound is never much looser than the simple microchoice bound because under the assumption that choice sets are of size at least $2$, the penalty for using the adaptive version, $lnd$, is always small compared to the complexity term for the simple microchoice bound, ${\sum }_{i=1}^{d\left(h\right)}ln\mid C_i\left(h\right)\mid$.

5.3.7. Other Adaptive Microchoice

The adaptive microchoice bound provides a simple scheme for dividing confidence between choices and queries. There are other choices which may be useful in some settings. Any scheme which a priori divides the confidence between queries and choices at every node will generally work. Here are two schemes which may be useful:

• Assign a constant proportion of confidence to the query. This scheme is more aggressive than the one used in the adaptive microchoice bounds and may result in a lower complexity when many choices are eliminatable. The drawback is we no longer get the telescoping in equation ( 5.3.1) and so the term logarithmic in $d\left(h\right)$ in theorem ( 5.3.4) becomes linear in $d\left(h\right)$.
• For a decision tree, assign a portion dependent on the depth of the node in the decision tree that the choice set is over. It is unlikely that choices are eliminatable from nodes not near the root because the number of examples available at a node typically decays exponentially with the (decision tree) depth. A progressive scheme which allocates less confidence to queries for deep nodes will probably behave better in practice.

5.3.8. Comparison with Freund’s Self-Bounding algorithms

Freund’s approach for self-bounding learning algorithms can require exponentially more computation then the microchoice approach. In its basic form, it requires explicit construction of every path in the state space of the algorithm not pruned in the tree. There exist some learning algorithms where this process can be done implicitly making the computation feasible. However, in general this does not appear to be possible.

The adaptive microchoice bound only requires explicit construction of the size of each subset from which a choice is made. Because many common learning algorithms work by a process of making choices from small subsets, this is often computationally easy. The adaptive microchoice bound does poorly, however, when Freund’s query tree has a high degree of sharing; for example, when many nodes of the tree correspond to the same query, or many leaves of the tree have the same final hypothesis. Allowing batch queries alleviates the most egregious examples of this. It is also possible to interpolate between the adaptive microchoice bound and Freund’s bound by a process of conglomerating the subsets of the microchoice bound.

5.3.9. Choice Set Conglomeration

The mechanism of choice set conglomeration is a similar to the batch query technique. It allows you to trade increased computation for a tighter bound. When starting with the simple microchoice bound, this technique can smoothly interpolate with the discrete hypothesis bound ( 4.2.1). When starting with the adaptive microchoice bound, we can interpolate with Freund’s bound.

Consider a particular choice set, ${\widehat{C}}_i$, with elements $c_i$. Each $c_i$ indexes another choice set, ${\widehat{C}}_{i+1}\left(c_i\right)$. If the computational resources exist to calculate the union ${\widehat{C}}_{i,i+1}={\bigcup }_{c_i\in {\widehat{C}}_i}{\widehat{C}}_{i+1}\left(c_i\right)$, then $\mid {\widehat{C}}_{i,i+1}\mid$ can be used in place of $\mid {\widehat{C}}_i\mid \cdot \mid {\widehat{C}}_{i+1}\mid$ in the adaptive microchoice bound. The conglomeration can be done repeatedly to build large choice sets and also applies to the simple microchoice bound ( 5.2.2). Conglomeration can be useful for tightening the bound when there are multiple choice sequences leading to the same hypothesis. However, choice set conglomeration is not always helpful because it trades away the fine granularity of the microchoice bound. The extreme case where all choice sets are conglomerated into one choice set and every hypothesis and query have the same weight is equivalent to Freund’s bound.

When the choices of the attached choice sets are all different, conglomeration will have little use because the size of the union of the choice sets is the sum of the sizes of each choice set $\mid {\widehat{C}}_{i,i+1}\mid ={\sum }_{c_i\in {\widehat{C}}_i}\mid {\widehat{C}}_{i+1}\left(c_i\right)\mid$. If the child sets each have the same size $\mid {\widehat{C}}_{i+1}\mid$ then this simplifies to $\mid {\widehat{C}}_{i,i+1}\mid =\mid {\widehat{C}}_i\mid \cdot \mid {\widehat{C}}_{i+1}\mid$ which results in the same confidence applied to each choice whether conglomerating or not. The best case for conglomeration is equivalent to the batch query case: every sub-choice set contains the same elements. Then we have $\mid {\widehat{C}}_{i,i+1}\mid =\mid {\widehat{C}}_{i+1}\mid$ and can pay no cost for the choice set $\mid {\widehat{C}}_i\mid$.