Chapter 7
Averaging Bounds (Improved margin)

The work in this chapter is joint with Matthias Seeger and Nimrod Megiddo. It was first presented at ICML [34].

Averaging bounds are specialized for averaging classifiers. An average has the form $$f\left(x\right)=\int q\left(h\right)h\left(x\right)dx\text{ or }f\left(x\right)={\int }_{i=1}^{N}q\left(h_i\right)h_i\left(x\right)dx$$ where $q\left(h_i\right)\ge 0$ and $\int q\left(h\right)dh=1$. Averaging classifiers have the form: $$c\left(x\right)=\text{sign}\left(f\left(x\right)\right)$$ Averaging bounds are especially interesting because there are many learning algorithms which use averaging. These techniques include:

1. Boosting [18]
2. Bayes-Optimal learning (see section 6.7 of [37])
3. Support Vector Machines [8]
4. Bagging [6]
5. Maximum Entropy classification [24]

Viewed as an interactive proof of learning (see figure 7.0.1) the bound presented here is almost the same as the PAC-Bayes bound except that it applies to the average over the posterior rather than to stochastic choices over the posterior.

The bound in this section is a qualitative improvement on prior results for averaging bounds. For the average learning algorithms listed above, the form of the improvements is most interesting for Maximum Entropy Classification and Bayes-Optimal classification. All (currently known) specialized averaging bounds use as a parameter the “margin”. For this section only, suppose the label and the hypotheses have value $-1$ or $1$. ($y\in \left\{-1,1\right\}$ and $h\left(x\right)\in \left\{-1,1\right\}$). Then, the margin will be defined as $$t\left(x,y\right)=yf\left(x\right)$$ Some simple observations are immediate:

1. The margin is bounded. $t\left(x,y\right)\in \left[-1,1\right]$
2. If the classifier is correct, the margin is positive. $c\left(x\right)=y\Rightarrow t\left(x,y\right)\in \left(0,1\right]$

The error at some margin is the quantity actually used in the averaging bounds. The empirical error at margin $\theta$ is defined by: $${\widehat{e}}_{\theta }\left(c\right)={Pr}_S\left(t\left(x,y\right)\le \theta \right)$$ and the true error at margin $\theta$ is defined by: $$e_{\theta }\left(c\right)={Pr}_{D^m}\left(t\left(x,y\right)\le \theta \right)$$ Note that $e_0\left(c\right)=e_D\left(c\right)$ (the true error rate of $c$).

The “margin” is a useful way to parameterize our learning algorithms. It will turn out that the sample complexity will be low (and the guarantees we can make better) when the margin of most of the training examples is large. There is, however, a price associated with using the margin: some hypotheses have no notion of a margin. Thus the theory in this chapter is less general than appears elsewhere.