12.2. Bound Application Details

Before introducing the results, we will mention a few important details about this bound.

12.2.1. Structural Risk Minimization

A naive application of the Shell bound would not prove useful because the size of the hypothesis space can be extremely large. Instead, we must combine it with Structural Risk Minimization (SRM) to achieve useful results. In SRM, you start with a bound for each hypothesis space, $H_i$, in a sequence of nested hypothesis spaces, $H_1\subseteq \cdots \subseteq H_n$. These bounds on individual hypothesis spaces are combined to create a bound which applies for all hypothesis spaces. The particular nesting we use is “$H_i=$ all decision trees with $i$ or fewer internal nodes”.

Since the size of the decision tree hypothesis spaces increases exponentially with the index $i$ , we choose $p_i=\frac{1}{2^i}$. This choice has the property that $\frac{1}{p_i}$ is always small in comparison to $\mid H_i\mid$, implying that the SRM bound is never much worse than a simple application of the underlying bound.

12.2.2. Computation

The computational cost of calculating some of these bounds is nontrivial. There are two basic parts to this computation;

1. Gathering the information in order to calculate the bound. This is sometimes infeasible for shell bounds and (later) PAC-Bayes bounds.
2. Combining the information in order to calculate the bound. For the shell bound, the amount of computation is like $O\left(m^{1.5}\right)$ where $m$ is the number of training examples. For the combined test and shell bound, the computation is approximately $O\left(m^2\right)$ .

We typically avoid the difficulties inherent in (1) using tricks such as monte carlo sampling followed by bounding the deviation of the monte carlo sample. In particular, we use the anytime computation trick of section 12.1.3 here.

To avoid difficulties inherent in problem (2), we use fast bounds (see section 3.2) on the Binomial tail as necessary.