7.1 Earlier Results

7.1. Earlier Results

The improved averaging bound arises from improving one critical step in the proof of the original margin bound [46] which is stated next.

THEOREM 7.1.1.

(Margin Bound [46]) For all $δ \in (0, 1]$ , for all base hypothesis spaces, $H$ , ${Pr}_{D^{m}} (\exists c, θ \in (0, 1] : e (c) > {\hat{e}}_{θ} (c) + O (\sqrt{\frac{\frac{ln ∣ H ∣}{θ^{2}} ln m + ln \frac{1}{δ}}{m}})) \leq δ$

PROOF. Given in [46]. A simplification of the improved averaging bound proof. ▫

Here, the notation $b (m) = O (a (m))$ means there exists a constant $C$ such that $b (m) \leq C \cdot a (m)$ for all $m$ . This margin bound implies that if most training examples have a large margin $θ$ (i.e. $t (x, y) > θ$ for most $(x, y) \in S$ ) and the hypothesis space is not too large, then the generalization error cannot be large. This theorem can only be non-vacuous when the base hypothesis space is finite. There are various extensions (see [46]) of this bound for continuous hypothesis spaces based upon VC dimension and covering number techniques. However, the extensions tend to result in extremely loose guarantees and are not relevant to the discussion here. One of the advantages of the improved averaging bound is that it can apply in a non-vacuous way to infinite hypothesis spaces. This generalization comes about with essentially zero loosening of the underlying bound.

[next] [front] [up]