12.1 The Decision Tree Learning Algorithm

12.1. The Decision Tree Learning Algorithm

Figure 12.1.1:

A plot of the sizes over various learning problems in terms of the number of examples. The learning problems used range over 3 orders of magnitude in size.

The bounds are applied to the results of decision trees learned on UCI database problems (see figure 12.1.1). The decision tree algorithm is a variant of ID3 which works in two phases: a tree of minimum empirical error is discovered and then pruned with a criteria which arises naturally from the Microchoice bound. The first phase works in a recursive fashion by maximizing the information gain across all splitting criteria. It is assumed that each example consists of $n$ features each of which takes a small number of discrete values. The splitting criteria at each internal node are of the form “feature $f$ has value $v$ ” which implies that a binary tree is produced. Leaves of the tree are labeled with the label most common amongst the examples in the training set which reach the leaf. The pruning pass prunes according to the criterion: minimize the upper bound on true error implied by the Microchoice bound. Pruning starts with the internal nodes closest to the leaves and proceeds up the tree toward the root node. This pruning criteria falls in the category of pessimistic criteria [38]. The implementation has been somewhat computationally optimized by careful caching of information. The examples are tokenized into integers for fast comparison and analysis of splitting criteria at each node only loops over the examples once. Sub-calculations of the Microchoice bound are cached for use in pruning.

12.1.1. Pruning

The proof of the Microchoice bound works by assigning a uniform weight to each choice in a choice space. Since the choice space of every node includes the choice of making a leaf, the Microchoice bound incidentally also proves a tighter bound for every pruning of the output tree. We choose to prune when pruning reduces the upper bound on the generalization error. We prune starting with internal nodes nearest to the leaves and working toward the root node. As will be shown by the experiments, other bounds are sometimes tighter than the Microchoice bound. This implies that the Microchoice bound is not always the optimal pruning criteria. However, the Microchoice bound is fast to calculate so it is used here. In practice, it may be desirable to prune according to the criteria “minimize $α {\hat{e}}_{S} (h) + (1 - α) b e (h)$ ” where $\bar{e} (h)$ is some high probability upper bound on the true error $e_{D} (h)$ . We use $α = 0$ with the bound produced by the Microchoice bound.

12.1.2. Uniform Sampling from Decision Trees

To use the Sampling Shell bound with Structural Risk Minimization, we need to sample uniformly from the set of all decision trees of a particular size. The number of binary structures of size $i$ is the $i$ th Catalan number, $C_{i} = \frac{1}{i + 1} (\binom{2 i}{i})$ . This implies there are $C_{i}$ different tree structures with $i$ internal nodes. Sampling uniformly from the set of all tree structures with $i$ internal nodes can then be done with a recursive process. For a particular node in a tree, assume that $j$ internal nodes need to be created. If $j = 0$ , we have a leaf. For $j \geq 1$ , we can construct a distribution over the number of nodes that are left branches of the tree by calculating $C_{0}, \dots, C_{j - 1}$ and normalizing. Draw from this distribution to get $j_{l}$ : the number of internal nodes in the left sub-tree. Let $j_{r} = j - j_{l} - 1$ be the number of internal nodes in the right sub-tree and recurse. This construction is not yet a decision tree because we have not placed tests in each node. We make another uniform pick from the set of available tests at a particular node. We allow only tests on features with at least two different feature values remaining after tests by parent nodes. This approach is not quite uniform in the set of decision trees because when there are $i$ internal nodes and $F < i$ Boolean features, some trees have a depth greater than $F$ which is not possible in a binary decision tree. In practice this does not make a difference because the number of decision trees with a too-large depth is an exponentially small fraction of the total number of trees. When running the algorithm we did not ever encounter a sampled binary tree structure with depth greater than $F$ . Leaves have a label picked uniformly from the set of labels. A particular label implies some empirical error count amongst all examples which reach the leaf. By adding the leaf error rates together we find the empirical error of a random hypothesis from the hypothesis set.

12.1.3. Fast Sampling

In order for the Sampling Shell bound to be a significant improvement we need the number of samples $l$ to satisfy $l = O (m)$ . This is not tractable using the above sampling technique so we “cheated”. At some critical sub-tree size $s$ we switch from sampling to exact enumeration. (The exact value of $s$ is discussed later.) Exact enumeration produces a multiset with elements corresponding to an error rate ${\hat{e}}_{i}$ in the sub-tree and a count $c_{i}$ associated with each element. We recurse up the tree with the entire error multiset rather than just one error. At an internal node we have a multiset of errors from the left child and from the right child. The multiset of errors at an internal node is the cross product of all errors in the left child and right child because the choice of left sub-tree is independent of the choice of right sub-tree. Let $S_{l}$ be the multiset of errors in the left child and $S_{r}$ be the multiset of errors in the right child. The multiset of errors for the internal node is then given by: $\{{\hat{e}}_{i} + {\hat{e}}_{j}, c_{i} * c_{j} : {\hat{e}}_{i}, c_{i} \in S_{l}, {\hat{e}}_{j}, c_{j} \in S_{r}\}$ The multiset produced is passed recursively to the parent and each element of the set of errors at the root node is considered an “independent” sample for the purposes of applying the Sampling Shell bound.

The power of this techniques comes from the fact that the set of hypotheses sampled is exponential in the number of leaves. However, we never need to represent the exponentially many different hypotheses explicitly because there are only $m + 1$ possible empirical errors. Using multisets, we achieve sampling time similar to the simple uniform sampling approach of the previous section but with exponentially more (dependent) samples. The drawback to this approach is that the samples are no longer independent so it is not “fair” to use them as independent samples in a theoretical sense. We pretend each fast-sample is independent anyway and apply the Sampling Shell bound. It is worth noting that the samples returned by this technique are not biased and sometimes have lower variance than independent samples.

The choice of the critical subtree size $s$ is done in an anytime fashion. The time $t$ required for learning the decision tree is first calculated. Then, starting with a size of $s = 0$ , we sample from the decision tree, increasing the size by one after each sample. When more than $t / 10$ time has been spent on sampling, we cease incrementing $s$ . If $s$ is the size of the tree, then we stop and apply the exact Shell bound. Otherwise, we decrement $s$ and sample repeatedly until as much time has been spent on sampling as was spent on learning the decision tree. The union of all the multisets is returned and the Sampling Shell bound is used.

[next] [front] [up]