10.2 Variance Analysis

Bounding the deviation of

{\hat{e}}_{pv}

is more difficult than bounding the deviation of a holdout error. To understand this, we can think of two games, the holdout game and the progressive validation game.

In the holdout game, your opponent chooses a bias and then nature flips

m

coins with that bias. If the deviation of the average number of heads is larger than

ε

, then you lose. Otherwise, you win.

In the progressive validation game, the opponent chooses the bias of each coin just before it is flipped. The goal of the opponent remains the same, and the opponent wins if a large deviation is observed.

The progressive validation opponent is at least as strong as the holdout opponent since the progressive validation opponent could choose the same bias for every coin. Nonetheless, we will see that the progressive validation opponent is not much stronger.

There are two ways in which we can show that the progressive validation opponent is not much stronger. The first technique will show that the variance of the progressive validation estimate is smaller than might be expected. Then we will show that the deviations of the progressive validation opponent behave much like the deviations of an independent opponent.

PROOF. Every example on the left hand side can be though of as a coin with bias $e_{pv} .$ The variance of the LHS is then $m * e_{pv} (1 - e_{pv})$ . The right hand side is: $E_{{(x, y)}^{m} \sim D^{m}} {({\hat{e}}_{pv} - e_{pv})}^{2}$ $= E_{{(x, y)}^{m} \sim D^{m}} {[\sum_{i = 1}^{m} {\hat{e}}_{i} - e_{i}]}^{2}$ where $e_{i} = e (h_{i})$ $= E_{{(x, y)}^{m} \sim D^{m}} [\sum_{i \neq j} ({\hat{e}}_{i} - e_{i}) ({\hat{e}}_{j} - e_{j}) + \sum_{i = 1}^{m} {({\hat{e}}_{i} - e_{i})}^{2}]$ $= \sum_{i \neq j} E_{{(x, y)}^{m} \sim D^{m}} ({\hat{e}}_{i} - e_{i}) ({\hat{e}}_{j} - e_{j}) + \sum_{i = 1}^{m} E_{{(x, y)}^{m} \sim D^{m}} {({\hat{e}}_{i} - e_{i})}^{2}$ The cross product term is: $E_{{(x, y)}^{m} \sim D^{m}} ({\hat{e}}_{i} - e_{i}) ({\hat{e}}_{j} - e_{j})$

Without loss of generality, assume that $i < j$ . What we wish to prove is that the expected value of this quantity is $0$ conditional on the values of all random variables other than $i$ th or $j$ th example. Let $S_{i j}$ be the set of examples minus the $i$ th and $j$ th example. Also, let $z_{i}$ and $z_{j}$ be the $i$ th and $j$ th labeled examples. If we can show that: $\forall S_{i j} : E_{z_{i}, z_{j} ∣ S_{i j}} ({\hat{e}}_{i} - e_{i}) ({\hat{e}}_{j} - e_{j}) = 0$ then linearity of expectation will imply that: $E_{{(x, y)}^{m} \sim D^{m}} ({\hat{e}}_{i} - e_{i}) ({\hat{e}}_{j} - e_{j}) = 0$

The value of $e_{i}$ is fixed after conditioning on $S_{i j}$ (and assuming a deterministic learning algorithm) while the value of $e_{j}$ is not fixed: it is dependent on the random variable $z_{i}$ . Let $e_{j} (z_{i})$ be the derived random variable. Then, the expectation is: $E_{z_{i}, z_{j} ∣ S_{i j}} ({\hat{e}}_{i} - e_{i}) ({\hat{e}}_{j} - e_{j})$ (implicitly conditioning on $S_{i j}$ ) $= \sum_{z_{i}, z_{j}} {Pr}_{D} (z_{i}) {Pr}_{D} (z_{j}) ({\hat{e}}_{i} (z_{i}) - e_{i}) ({\hat{e}}_{j} (z_{j}, z_{i}) - e_{j} (z_{i}))$ $= \sum_{z_{i}} {Pr}_{D} (z_{i}) ({\hat{e}}_{i} (z_{i}) - e_{i}) \sum_{z_{j}} {Pr}_{D} (z_{j}) ({\hat{e}}_{j} (z_{j}, z_{i}) - e_{j} (z_{i}))$ For any fixed $z_{i}$ , we want to show that $\sum_{z_{j}} {Pr}_{D} (z_{j}) ({\hat{e}}_{j} (z_{j}, z_{i}) - e_{j} (z_{i})) = 0$ .With $z_{i}$ fixed, the true error rate of the $j$ th hypothesis is $e_{j} (z_{i})$ . Therefore, the probability of observing an error ( ${\hat{e}}_{j} (z_{j}, z_{i}) = 1$ ) is $e_{j} (z_{i})$ , and the probability of observing no error ( ${\hat{e}}_{j} (z_{j}, z_{i}) = 0$ ) is $1 - e_{j} (z_{i})$ . This implies: $\sum_{z_{j}} {Pr}_{D} (z_{j}) ({\hat{e}}_{j} (z_{j}, z_{i}) - e_{j} (z_{i}))$ $= \sum_{z_{j} : {\hat{e}}_{j} (z_{j}, z_{i}) = 0} {Pr}_{D} (z_{j}) (- e_{j} (z_{i})) + \sum_{z_{j} : {\hat{e}}_{j} (z_{j}, z_{i}) = 1} {Pr}_{D} (z_{j}) (1 - e_{j} (z_{i}))$ $= (1 - e_{j} (z_{i})) (- e_{j} (z_{i})) + e_{j} (z_{i}) (1 - e_{j} (z_{i}))$ $= 0$

Putting this together, we get: $cross term = \sum_{z_{i}} {Pr}_{D} (z_{i}) ({\hat{e}}_{i} (z_{i}) - e_{i}) * 0$

$= 0$ Since this sum is zero regardless of the values of all other random variables, the cross product terms all have expectation zero. Note that we can extend this proof to randomized algorithms by conditioning on the random bits of the algorithm in the above argument. Consequently: $= \sum_{i = 1}^{m} E_{{(x, y)}^{m} \sim D^{m}} [{({\hat{e}}_{i} - e_{i})}^{2}]$ $= \sum_{i = 1}^{m} e_{i} (1 - e_{i})$ So, all that we must show is: $m e_{pv} (1 - e_{pv}) \geq \sum_{i = 1}^{m} e_{i} (1 - e_{i})$ which follows from Jensen’s inequality and the convexity of $x (1 - x)$ on the interval $[0, 1]$ . ▫

10.2. Variance Analysis