4.5. Structural Risk Minimization

Structural Risk Minimization [51] (SRM) is a technique used in the learning theory community to avoid the difficulties associated with convergence on hypothesis sets that are too “large”. SRM works with a sequence of nested hypothesis sets, $H_1\subset H_2\subset ....\subset H_l$. For each hypothesis set, a discrete hypothesis bound ( 4.2.1) on the difference between empirical and true error exists. For “small” hypothesis sets, this bound may be tight while for large hypothesis sets it may be inherently loose. However, we also expect that the best hypothesis in the hypothesis set improves as the hypothesis set becomes larger. This naturally induces a trade-off: there will be some hypothesis set $H_i$ for which the true error bound is minimized.

We can’t simply apply the discrete hypothesis bound to the meta-algorithm which picks the algorithm (and associated hypothesis space) with the smallest true error bound since this meta-algorithm could, potentially, output any hypothesis in $H_l$. The simplest way to retrofit the bound to include all hypothesis sets is with a simple theorem which essentially states that we can guarantee nearly the same bound as would apply on the smallest hypothesis space $H_i$ containing the output hypothesis, $h$.

The SRM bound is slightly inefficient in the sense that the bound for all hypotheses in $H_2$ includes a bound for every hypothesis in $H_1$. This effect is typically small because the size of the hypothesis sets usually grows exponentially, implying that the extra confidence given to a hypothesis $h$ in $H_1$ by the bounds used on hypothesis set $H_2,H_3,...$ is small relative to the confidence given by the bound for $H_1$. One can remove this slack in Structural Risk Minimization bound by “cutting out” the nested portion of each hypothesis set in the formulation of $H_1,...,H_l$. We will call this Disjoint Structural Risk Minimization (also mentioned in [?]).