11.3 Approximations in Combinations

The inexact nature of bounds forces us to impose a monotonic structure on the function

f (S_{test})

. For simplicity, we will restrict to functions of the form

f ({\hat{e}}_{test} (h))

where

{\hat{e}}_{test} (h)

is the test error on hypothesis

h

. This simplification is not necessary and this technique can be extended to arbitrary test set based techniques.

We can consider any upper bound

θ_{D} (δ)

on the true error rate,

e_{D} (h)

, as inducing a cumulative distribution on the test set events.

This cumulative distribution is not the cumulative distribution of the underlying (Binomial) probability. To construct this distribution, let:

F_{θ} ({\hat{e}}_{test} (h)) = inf {δ : {\hat{e}}_{test} (h) \in θ_{D} (δ)}

Intuitively,

F_{θ} ({\hat{e}}_{test} (h))

is the smallest

δ

such that the test error

{\hat{e}}_{test} (h)

is rejected.

LEMMA 11.3.1.

The function $F_{θ} ({\hat{e}}_{test} (h))$ is a cumulative distribution function.

PROOF. In order to show that the function is a cumulative distribution function, we must show that it varies between $0$ and $1$ for all values of ${\hat{e}}_{test} (h)$ . Since $θ_{D} (δ)$ is an upper bound, the following inequality holds: $\forall S_{test} F_{θ} ({\hat{e}}_{test} (h)) \geq Bin (m, m * {\hat{e}}_{test} (h), e_{D} (h))$ This inequality implies the value of $F_{θ} ({\hat{e}}_{test} (h))$ is always at least as large as the CDF of the underlying Binomial distribution. Note, that different $S_{test}$ are implicitly aliased under this technique. We also have the inequality $F_{θ} ({\hat{e}}_{test} (h)) \leq 1$ because all true error rate upper bounds are vacuous above a true error rate bound of $1$ . ▫

We have shown that

F_{θ}

is a cumulative distribution function over the value of the empirical error. Given the upper bound cumulative,

F_{θ}

, we can look at distributions satisfying:

E_{{\hat{e}}_{test} (h) \sim F_{θ}} {Pr}_{S \sim D^{m}} (S \in φ (f ({\hat{e}}_{test} (h)) * δ)) \leq δ

If we are guaranteed that

f ({\hat{e}}_{test} (h))

decreases monotonically then equation 11.2.2 will hold. This is the essence of our theorem.

THEOREM 11.3.2.

(Approximate test and train bound) Let $φ_{D} (δ)$ be any bound satisfying ${Pr}_{S \sim D^{m}} (S \in φ_{D} (δ)) \leq δ$ . Let $f ({\hat{e}}_{test} (h))$ be any monotonic decreasing function satisfying: $E_{{\hat{e}}_{test} (h) \sim F_{θ}} {Pr}_{S \sim D^{m}} (S \in φ_{D} (f ({\hat{e}}_{test} (h)) * δ)) \leq δ$ , then: ${Pr}_{S, S_{test} \sim D^{m + m_{test}}} (S \in φ_{D} (f ({\hat{e}}_{test} (h)) * δ)) \leq δ$

PROOF. Note that ${Pr}_{S \sim D^{m}} (S \in φ_{D} (f ({\hat{e}}_{test} (h)) * δ))$ is a monotonic decreasing function of ${\hat{e}}_{test} (h)$ . For any monotonic decreasing function $g (x)$ and any two cumulative distribution functions $F_{1} (x)$ and $F_{2} (x)$ satisfying $\forall x F_{1} (x) \leq F_{2} (x)$ we have: $E_{x \sim F_{1} (x)} g (x) \leq E_{x \sim F_{2} (x)} g (x)$ Let $F (x)$ be the cumulative distribution of the Binomial and note that the definition of a bound implies: $\forall x F_{θ} (x) \geq F (x)$ . Applying these inequalities, we get: $δ \geq E_{{\hat{e}}_{test} \sim F_{θ}} {Pr}_{S \sim D^{m}} (S \in φ_{D} (f ({\hat{e}}_{test} (h)) * δ))$ $\geq E_{{\hat{e}}_{test} \sim F} {Pr}_{S \sim D^{m}} (S \in φ_{D} (f ({\hat{e}}_{test} (h)) * δ))$ Given this, an application of theorem 11.2.1 completes the proof. ▫

The only constraint that we must check in applying a combined training and test set bound is the monotonicity constraint. Heuristically, this is satisfied for the functions graphed in the pictures of techniques (1)-(3) because the set of excluded events increases monotonically along the

x

axis as it decreases along the

y

axis.

An explicit mathematical form for technique (3) can be given by considering the bound-based cumulative distributions,

F_{θ}

, and a similar distribution for the training set,

F_{φ}

. In particular, we can define the rejection region to be

{S_{test}, S : F_{θ} F_{φ} \leq t (δ)}

where

t (δ)

is a function satisfying

{Pr}_{S_{test}, S \sim F_{θ}, F_{φ}} (F_{θ} ({\hat{e}}_{test} (h)) F_{φ} (S)) \leq t) \leq δ

. The monotonic constraint is satisfied by this construction because

F_{θ}

is implicitly monotonic decreasing with

F_{φ} (S)

given a constant

t

. We will use technique (3) for combining training and test sets in the experiments. Appendix section 16.4 details the programming interface to calculate this bound.

11.3. Approximations in Combinations