11.2. General Approaches for Combined Bounds

Showing that a more general technique works must start with a discussion of confidence intervals. Fundamentally, a bound can be viewed as a set of outcomes. Let $X$ be space of outcomes, then a bound $\phi \subseteq X$ is a subset. The probability that this generalized bound is violated is given by: $${Pr}_{x\sim P}\left(x\in \phi \right)$$ Typically, we parameterize $\phi$ with both $P$ and $\delta$ to get: $${Pr}_{x\sim P}\left(x\in {\phi }_P\left(\delta \right)\right)\le \delta$$ We can expand the definition of a high probability set to include a randomized high probability set. In particular, let ${\phi }_P\left(w,\delta \right)$ satisfy: $$\forall w:{Pr}_{x\sim P}\left(x\in {\phi }_P\left(w,\delta \right)\right)\le \delta$$ Then, statement such as: $${Pr}_{w\sim Q,x\sim P}\left(x\in {\phi }_P\left(w,\delta \right)\right)\le \delta$$ In fact, we can make a stronger statement. If

Randomized confidence intervals are useful here because we can regard the the draw of the “test” set as constructing a randomized interval for the “training” set. As long as constraint 11.2.1 is obeyed, the bound will hold with probability at least $\delta$. A P-value Inversion of the bound will then yield the following theorem:

There are many possible choices of the function $f\left(S_{\text{test}}\right)$, each of which leads a different combined training and testing bound. Typically, it will be important to invert this bound into a P-value form where we take a worst case over all $D$ just as in section 3.4.

The functional form of $f\left(S_{\text{test}}\right)$ becomes more constrained when we only work with a bound on the test error cumulative distribution rather than the exact distribution.