7.2 A generalized averaging bound

Before discussing the main theorem, it is important to notice that the averaging classifier,

c (x)

implies a distribution over the base hypothesis space

H

. This implied distribution is

q_{c} (h)

where

c (x) = sign (\int h (x) q_{c} (h) d h)

The distribution

q_{c}

is used in the following theorem.

THEOREM 7.2.1.

(Relative Entropy Averaging Theorem) For all distribution $p (h)$ , for all $δ \in (0, 1]$ :

${Pr}_{D^{m}} (\exists c, θ \in (0, 1] : KL ({\hat{e}}_{θ} (c) ∣ ∣ e (c)) \geq O (\frac{\frac{KL (q_{c} ∣ ∣ p)}{θ^{2}} ln m + ln m + ln \frac{1}{δ}}{m})) \leq δ$

PROOF. Given in the next section. ▫

The main theorem uses a KL-divergence based pseudodistance which is a bit hard to understand intuitively. In order to gain intuition, we can relax the tightness of the proof with an inequality.

KL ({\hat{e}}_{θ} (c) ∣ ∣ e (c)) \geq 2 {({\hat{e}}_{θ} (c) - e (c))}^{2}

COROLLARY 7.2.2.

(Relative Entropy Averaging Theorem) For all distribution $p (h)$ , for all $δ \in (0, 1]$ :

THEOREM 7.2.3.

${Pr}_{D^{m}} (\exists c, θ \in (0, 1] : e (c) \geq {\hat{e}}_{θ} (c) + O (\sqrt{\frac{\frac{KL (q_{c} ∣ ∣ p)}{θ^{2}} ln m + ln m + ln \frac{1}{δ}}{m}})) \leq δ$

This theorem improves upon theorem 7.1.1 because

KL (q_{c} ∣ ∣ p)

is used instead of

ln ∣ H ∣

. For the case of a uniform distribution on

∣ H ∣

different base classifiers, these results will agree when the average is over just one classifier. As the average becomes “broader” the results will improve. In the limit when the average is over nearly all classifiers, the term

KL (q_{c} ∣ ∣ p)

will be nearly

0

The theorems are stated in an asymptotic fashion which is not be very useful in practical applications. Section 7.4 gives some ideas of how to tighten the result, and the non-asymptotic form ( 7.3.15) given at the end of the proof can be used directly in practice.

The improved averaging bound applies to averages over continuous hypothesis spaces. In this setting, the average needs to be an integral over an uncountably-infinite set of hypotheses or the KL-divergence will not converge to a finite value. It is exactly because of this limitation that the improvements of this bound are most applicable to Bayes Optimal and Maximum Entropy classifiers.

In practice, the limitation may not be a significant problem because machine learning algorithms over large hypothesis spaces typically have some parameter stability. In other words, a small shift in the parameters of the learned model produces a small change in the prediction of the hypothesis. With hypothesis stability, we can convert any average over a finite set of hypotheses into an average over an infinite set of hypotheses without significantly altering the predictions of the average. This technique is explored in chapter 13 with positive results.

7.2. A generalized averaging bound