12.3 Results & Discussion

12.3. Results & Discussion

12.3.1. Holdout bound

Figure 12.3.1:

This is a graph of the true error upper bound give by the holdout bound ( 4.1.1). In this figure (and all others) we use

δ = 0.1

probability of failure for the tail. Here “error” is the test error. The red lines can be interpreted as the region where the true error rate might exist given that we are in a high probability case.

Our goal is to bound the true error of the hypothesis output by our learning algorithm. To do this, we apply sample complexity bounds to the results of the decision tree on UCI database problems. The problems chosen from the UCI database are those for which a discrete decision tree is applicable. All bounds are calculated with a probability of failure of

δ = 0.1

. As mentioned in the introduction, there are two approaches. The commonly used approach is to first divide the example set into two sets,

m_{test}

and

m_{train}

. Then, train using the examples in

m_{train}

and test on the

m_{test}

examples. We chose an

80 / 20

split of the data into

m_{train} / m_{test}

. We will compare each bound with the simple holdout approach because this is the commonly used baseline.

12.3.2. Comparison with a standard confidence interval approach

Figure 12.3.2:

This is a graph of the confidence intervals implied by the holdout bound ( 4.1.1) on the left, and the approximate confidence intervals implied using the common two sigma rule motivated by asymptotic normality on the right. For this graph only, the upper and lower bounds of the holdout bound have a maximum

2.5 %

failure rate (each), rather than a

10 %

failure rate. This is done in order to make the results more comparable with the

2

-sigma approach. The holdout bound is better behaved in the sense that the confidence interval is confined to the interval

[0, 1]

and it is never over-optimistic.

When attempting to calculate a confidence interval on the true error rate given the holdout set, many people follow a standard statistical prescription:

Calculate the empirical mean $\hat{μ} = {\hat{e}}_{S_{test}} (h) = \frac{1}{m} \sum_{i = 1}^{m} I (h (x_{i}) \neq y_{i})$ .
Calculate the empirical variance ${\hat{σ}}^{2} = \frac{1}{m - 1} \sum_{i = 1}^{m} {(I (h (x_{i}) \neq y_{i}) - \hat{μ})}^{2}$ .
Pretend that the distribution is a normal with the above parameters and construct a confidence interval by cutting the tails of the Gaussian cumulative distribution.

This approach is motivated by the fact that for any fixed true error rate, the distribution of empirical errors will behave like a gaussian asymptotically. Here, asymptotically means “in the limit as the number of test examples goes to infinity”.

The problem with this approach is that it leads to fundamentally misleading results. In particular, 12.3.2 shows that the confidence interval is not confined to the interval $[0, 1]$ . It is difficult to give an interpretation to intervals with boundaries less than $0$ or greater than $1$ . In addition, this approach is sometimes highly overoptimistic. When the test error is $0$ , our confidence interval should not have size $0$ for any finite $m$ .

In contrast, the holdout bound approach uses the underlying Binomial distribution directly. This implies:

The holdout bound approach is never optimistic.
The holdout bound based confidence interval always returns an upper and lower bound in $[0, 1]$ .
The holdout bound approach is more accurate.

The bootstrap [15] is sometimes used as a confidence interval. The assumption under which this works is essentially equivalent to an assumption of “enough” data. For finite amounts of data, the bootstrap “confidence intervals” will necessarily be violated on datasets with phase transitions such as 10.4.1. This is discussed more in the next section.

12.3.3. Comparison with point estimators

Point estimators are techniques for directly estimating the value of the true error. In theory, there should be no need to compare point estimators with confidence interval bounds such as those discussed here because the goals are simply different: point estimators attempt to estimate the value of the true error while confidence intervals confine the value of the true error to an interval with high probability. However, point estimators are often used for more than estimating true error. It is a common practice to use point estimators in deciding which of two learning algorithms (or learning algorithm parameters) is better.

There are several several point estimators in use, including:

Holdout test set error rate.
The bootstrap.

One commonly used point estimator is the bootstrap. In typical use, the bootstrap which functions like this:

Repeat many times:

Pick $m$ examples uniformly from the set of $m$ examples.
Train on the $m$ examples
Test on examples not included in the training set.

After the above computation, the training and test errors are combined according some formula (which often varies) to get an estimate of the true error rate of a hypothesis learned on all of the $m$ original examples.

There is one immediate observation: the resampling process typically results in about $m (1 - \frac{1}{e})$ unique examples being included in the resampled subset. This has very strong implications because there exist learning problems with “phase transitions” where the accuracy of the learned hypothesis (even for the best possible learning algorithm) as a function of $m$ decreases suddenly when $m$ reaches some critical threshold. This implies that point estimators cannot always be accurate on dataset with a phase transition like 10.4.1. When learning a hypothesis on $m (1 - \frac{1}{e})$ examples results in a true error rate of $0.5$ , it could be the case that learning on $m$ examples results in a true error rate of $0$ or it could be the case that the true error rate will be $0.5$ .

Given that the bootstrap can sometimes fail to predict the true error rate, reasoning about which algorithm is preferable based upon the bootstrap output is questionable. One alternative to this is reasoning with the criteria:

Pick the learning algorithm with the lower upper bound.

Assuming that examples are independent, this approach can never fail arbitrarily badly (with high probability).

There are still questionable issues with this approach such as: “What if the upper bound is not tight?” It could be the case that a better learning algorithm has a worse upper bound implying that the worse algorithm will be picked according to this criteria. One solution to this dilemma is to always involve some small amount of holdout examples in your bound calculation. Used judiciously, these holdout examples can guarantee that the bound-based criteria never becomes too loose.

12.3.4. Simplistic bounds vs. the Holdout bound

Figure 12.3.3:

This is a plot comparing confidence intervals built based upon the holdout bound ( 4.1.1) on the left and the simple training bound using the Hoeffding inequality ( 4.2.3) on the right. The results are clearly unsatisfactory for the simple training bound.

Figure 12.3.3 compares a bound based upon theorem 4.2.3 and theorem 4.1.1. It is remarkably pessimistic about the prospect of training set based bounds because the confidence intervals are essentially vacuous. This bound can be improved always by using exact (rather than approximate) calculations of the Binomial tail. It can also be improved in practice by using a nonuniform “prior” over the hypothesis space.

12.3.5. Occam vs. the Holdout bound

Figure 12.3.4:

This is a plot comparing confidence intervals built based upon the holdout bound ( 4.1.1) on the left and an “Occam’s Razor” style bound ( 4.6.1) using the Hoeffding inequality ( 4.2.3) on the right. This training set bound is sometimes useful on the very small datasets. The particular description length was built using a microchoice-like description language.

A “prior” (in the sense of theorem 4.6.1) does not help much with the visible confidence intervals, although an examination of the calculations suggests that improvements do exist - they just aren’t enough to make the confidence intervals nonvacuous in figure 12.3.4. Note that the “prior” used here is the Microchoice prior. Next, we will get rid of the approximation.

12.3.6. Microchoice

Figure 12.3.5:

This is a plot comparing confidence intervals built based upon the holdout bound ( 4.1.1) on the left and the microchoice bound ( 5.2.2) on the right. The most significant improvement over the last bound is using a Binomial tail bound calculation rather than the Hoeffding approximation. The effect of this improved bound is quite significant when the training error is low. We achieve a tighter upper bound on 4 learning problems.

For the first time, we observe confidence intervals which are nonvacuous on a training set in figure 12.3.5. This is encouraging, and a comparison with the holdout approach indicates that the training set based confidence intervals are actually superior on datasets with a small number of examples (and thus with a very small holdout set).

12.3.7. Shell Bound

Figure 12.3.6:

This is a plot comparing confidence intervals built based upon the holdout bound ( 4.1.1) on the left and the shell upper bound theorem ( 8.1.2 or 8.2.1) on the right. The light-dashed line indicates which of the results were obtained using the fast sampling technique. The shell-based upper bound is lower than the holdout upper bound on several of the 13 problems and is never significantly looser. Note that the shell bound is computationally intensive with

O (m^{1.5})

running time plus the time required to gather the necessary information.

The Shell bound performs better than the Microchoice bound in figure 12.3.6. The information and computation requirements needed to calculate the shell bound are quite large, but the resulting bound is noticeably tighter, especially on problems with more examples. This bound is strong evidence that training set based bounds can be made competitive with test set based bounds. However, it is unnecessary to choose between these approaches since we can construct a bound which uses information from both training and test set based bounds.

12.3.8. Combined Microchoice and holdout bound

Figure 12.3.7:

This is a plot comparing confidence intervals built based upon the holdout bound ( 4.1.1) on the left and a combination (theorem 11.2.1) of the holdout and microchoice (theorem 5.2.2) bounds. The middle column is the microchoice bound and the right column is the combined bound.

The resulting combined bound consistently performs well on all data sets and is sometimes superior to either bound individually. The computational time of this bound is

O (m^{1.5})

in general.

The combined Microchoice and Holdout bound performs only slightly worse than the best of the either bound and is sometimes better than either bound individually for the problems reported in figure 12.3.7. This particular combined bound is (perhaps) the most practical result of this thesis since it is easy to calculate the necessary information and reasonably easy to calculate the value of the bound.

12.3.9. Combined Shell and Holdout Bound

Figure 12.3.8:

This is a plot comparing confidence intervals built based upon the holdout bound ( 4.1.1) on the left and a combination (theorem 11.2.1) of the holdout (theorem 4.1.1) and shell bounds (theorem 8.1.2 and 8.2.1). The middle column is the shell bound or sampling shell bound (if a dashed line is present). The right column is the combined bound

The computational cost of this bound is very nontrivial taking

O (m^{2})

time in general.

The combined shell and holdout bound gives the best results of all in figure 12.3.8. The downside of using the shell bound is that significantly more computation and information is required in order to calculate the bound. The computational cost for the bound is

O (m^{2})

which makes it impractical to apply this bound beyond about

m = 10000

with current computers.

[next] [prev] [prev-tail] [front] [up]