Chapter 12
Decision Trees

Are Sample Complexity bounds quantitatively tight? The real challenge lies with continuous valued classifiers and will be addressed in the next chapter. Before worrying about continuous valued classifiers it is worthwhile to consider performance on a discrete valued classifier. To do this, we will apply a (discrete valued) decision tree to learning problems in the UCI machine learning database.

The results of this analysis are interesting - competitive bounds are achieved in some cases and the best sample complexity bounds are never more than an order of magnitude worse than a reasonable holdout based approach using the same resources.

Most practitioners of applied machine learning currently use a different technique for free parameter optimization: holdout sets. The simplest form of this technique is to separate the examples into a “training set” and “testing set”. The training set is used by the learning algorithm to output a hypothesis. The hypothesis is then tested on the test set to generate an estimate of the future error rate. The principal advantage of the holdout technique is the (simple theoretical) guarantee that with high probability the estimate will not be much higher or lower than the true error rate. Here the quantity “much” depends upon the size of the holdout set and the “high probability” can be chosen as desired by the person applying the bound.

The principal disadvantage of the holdout approach is that not all of the examples are used for training. This is often not a significant problem but it can be important for certain learning algorithms and problems which exhibit phase transitions (see figure 10.4.1 on page 276 for an example). Near a phase transition , extra examples can exponentially decrease the expected error rate of the output hypothesis. More sophisticated techniques such as Leave One Out Cross Validation, K-fold Cross Validation, and Progressive Validation [3] attempt to remove this disadvantage. Each of these alternative holdout techniques cannot fully remove the disadvantage and some of them threaten the advantage (a tight bound on the true error). In addition, a new disadvantage is introduced: significantly more computation is required.

The holdout technique is fundamentally unsatisfactory for one important reason: If a holdout set is used multiple times, the theoretical guarantees become progressively weaker. In particular, this implies that designing a learning algorithm which makes multiple internal decisions based upon the result of evaluating a future error bound for holdout sets may require many examples. In fact, enough examples are required that a “multiply used holdout set” is described by the same math as a training set based sample complexity bound.

We will compare the following bounds on a decision tree:

1. The discrete hypothesis bound 4.2.1.
2. The Occam’s Razor bound 4.6.1 with the Hoeffding Binomial tail bound. This bound might be thought of as an old “state of the art” bound.
3. The Microchoice bound 5.2.2. In addition, we will prune the decision tree according to the microchoice bound.
4. The Shell bound 8.1.2 and the Sampled Shell bound 8.2.1.
5. A simple holdout bound 4.1.1 using a holdout set of size $20\%$.
6. A combined training and testing bound using the holdout bound 4.1.1 and microchoice bound 5.2.2.
7. A combined training and testing bound using the holdout bound 4.1.1 and the (stochastic) shell bound ( 8.2.1) 8.1.2.

We will first discuss the decision tree and bound calculation implementation. Then present results