7.3 Proof of main theorem

7.3. Proof of main theorem

7.3.1. Definitions and observations

The proof has the same structure as the original margin bound ( 7.1.1) proof with one step replaced by the application of the relative entropy PAC-Bayes theorem ( 6.2.1).

Let $N$ be any natural number; later, the choice of $N$ will be optimized. In the first part of the proof, we regard $θ$ and $N$ as fixed. Later we generalize this so that they may depend on the sample $S$ .

We construct the distribution $q_{N}$ as follows. Draw $N$ hypotheses $h_{i} \sim q (h)$ independently. $Q_{N}$ might therefore be viewed as the product distribution

q_{c} {(h)}^{N} .

(7.3.1)

Given the

h_{i}

we define

g (x) = \frac{1}{N} \sum_{i = 1}^{N} h_{i} (x)

(7.3.2)

For fixed $x, y$ , the value of $y h (x)$ are i.i.d. Bernoulli variables with the mean equal to the expected margin:

E_{q \sim h} y g (x) = y f (x)

(7.3.3)

Therefore,

\forall x, y E_{g \sim Q_{N}} [y g (x)] = y f (x)

and

y g (x)

is distributed according to the familiar Binomial distribution. Later, we will apply a Binomial tail bound on this quantity.

Before writing out the proof mathematically, it is helpful to consider a graphical view of what we will prove. We will force convergence between three quantities.

The convergences are then tied together with triangle inequalities resulting in the overall proof. The critical improvement of this paper is a refined version of the second convergence.

7.3.2. The Proof

For every $θ \in (0, 1]$ and for every (fixed) $g$ , the following simple inequality holds:

e (f) = {Pr}_{D} [y f (x) \leq 0]

(7.3.4)

= {Pr}_{D} [y g (x) \leq \frac{θ}{2}, y f (x) \leq 0] + {Pr}_{D} [y g (x) > \frac{θ}{2}, y f (x) \leq 0] ∣ y f (x) \leq 0]

\leq {Pr}_{D} [y g (x) \leq \frac{θ}{2}] + {Pr}_{D} [y g (x) > \frac{θ}{2} ∣ y f (x) \leq 0]

Note that the left-hand side does not depend on

g

. By taking the expectation over

g \sim Q_{N}

(and exchanging the order of expectations in the second term on the right-hand side), we arrive at

e (f) \leq E_{g \sim Q_{N}} [{Pr}_{D} [y g (x) \leq \frac{θ}{2}]] + E_{D} [P_{g \sim Q_{N}} [y g (x) > \frac{θ}{2} ∣ y f (x) \leq 0]] .

(7.3.5)

For the rightmost term, we can apply a Binomial tail bound to get:

e (f) \leq E_{g \sim Q_{N}} [{Pr}_{D} [y g (x) \leq \frac{θ}{2}]] + E_{D} [1 - Bin (N, \delimiter N \frac{1 + θ / 2}{2}\delimiter , 0)]

(7.3.6)

= E_{g \sim Q_{N}} [{Pr}_{D} [y g (x) \leq \frac{θ}{2}]] + 1 - Bin (N, \delimiter N \frac{1 + θ / 2}{2}\delimiter , 0)

(7.3.7)

We would like to apply the PAC-Bayes theorem 6.2.1 to the right-hand term. Here we use the loss function

I_{{y g (x) \leq θ / 2}}

. The PAC-Bayes theorem applies for any “prior” distribution. We use as the “prior” the distribution

P_{N} = p {(h)}^{N}

where

p (h)

is the “prior” over the base hypothesis space. The choice of this prior allows us to use the identity:

KL (Q_{N} ∣ ∣ P_{N}) = N KL {q (h) ∣ ∣ p (h))

It follows from Theorem 6.2.1 that: with probability at least

1 - δ

over random choices of

S

, for every

Q

{Pr}_{D} (\exists q (h) : KL ({\hat{e}}_{\frac{θ}{2}} (g) ∣ ∣ e_{\frac{θ}{2}} (g)) \leq \frac{KL (Q_{N} ∣ ∣ P_{N}) + ln \frac{m}{δ}}{m - 1})

(7.3.8)

where

e_{\frac{θ}{2}} (g) = 1

with probability

E_{g \sim Q_{N}} [{Pr}_{D} [y g (x) \leq \frac{θ}{2}]]

and

0

otherwise, and

{\hat{e}}_{\frac{θ}{2}} (g) = 1

with probability

E_{g \sim Q_{N}} [{Pr}_{S} [y g (x) \leq \frac{θ}{2}]]

and

0

otherwise.

By the same argument as in ( 7.3.4), we get:

{\hat{e}}_{\frac{θ}{2}} (g) = {Pr}_{S} [y g (x) \leq \frac{θ}{2}] \leq {Pr}_{S} [y g (x) \leq \frac{θ}{2}, y f (x) > θ] + {Pr}_{S} [y f (x) \leq θ]

(7.3.9)

\leq {Pr}_{S} [y g (x) \leq \frac{θ}{2} ∣ y f (x) > θ] + {Pr}_{S} [y f (x) \leq θ]

(7.3.10)

= {Pr}_{S} [y g (x) \leq \frac{θ}{2} ∣ y f (x) > θ] + {\hat{e}}_{θ} (f)

(7.3.11)

Again, we take expectations over

g \sim Q_{N}

on both sides, interchange the order of the expectations and apply the Binomial tail bound to get:

E_{S} [P_{g \sim Q_{N}} [y g (x) \leq \frac{θ}{2}]] \leq Bin (N, \delimiter N \frac{1 + θ / 2}{2}\delimiter , \frac{1 + θ}{2}) + {Pr}_{S} [y f (x) \leq θ] .

(7.3.12)

Combining our inequalities, we conclude that with probability at least

1 - δ

, for every

q (h)

KL (q_{S} ∣ ∣ p_{D}) \leq \frac{N KL (q ∣ ∣ p) + ln \frac{m}{δ}}{m - 1}

(7.3.13)

where

q_{S} = 1

with probability

Bin (N, \delimiter N \frac{1 + θ / 2}{2}\delimiter , \frac{1 + θ}{2}) + {Pr}_{S} [y f (x) \leq θ]

and

0

otherwise, and

p_{D} = 1

with probability

{Pr}_{D} [y f (x) \leq 0] - 1 + Bin (N, \delimiter N \frac{1 + θ / 2}{2}\delimiter , 0)

and

0

otherwise.

This bound holds for any fixed $N$ and $θ$ , which is not yet what we need here, since we want to allow these to depend on the data $S$ . In essence, the bound we proved so far is a statement about certain events, parameterized by $N$ and $θ$ , namely the probability of each event is smaller than $δ$ . However, we need to prove that the probability of the union of all these events is smaller than $δ$ . To this end, we first observe that this union is contained in the union over only a countable number of events. Note that $g (x) \in {(2 k - N) / N ∣ k = 0, 1, \dots, N}$ . Thus, even with all the possible (positive) values of $θ$ , there are no more than $N + 1$ events of the form ${y g (x) \leq θ / 2}$ . Denote by $k (θ, N)$ the largest integer $k$ such that $k / N \leq θ / 2$ . We observe that for every $θ > 0$ , every $g$ and every distribution over $(x, y)$ :

Pr [y g (x) \leq θ / 2] = Pr [y g (x) \leq \frac{k (θ, N)}{N}] .

(7.3.14)

This means that the middle step in the proof above, i.e. the application of theorem 6.2.1, depends on

(N, θ)

only through

(N, k)

. Since the other steps are true with probability one, we see that we can restrict ourselves to the union of countably many events, indexed by

(N, k)

. Now, we “allocate” parts of the confidence quantity

δ

to each of these events, namely

(N, k)

receives

δ_{N, k} = δ / (N {(N + 1)}^{2}), N = 1, 2, \dots; k = 0, \dots, N

. It follows easily that the union of all these events has probability at most

\sum_{N, k} δ_{N, k} = δ

. Therefore we have proved that with probability at least

1 - δ

over random choices of

S

it holds true that for all

N

and all

θ \in (0, 1]

KL (q_{S} ∣ ∣ p_{D}) \leq \frac{N KL (Q ∣ ∣ P) + ln \frac{2 m}{δ_{N, k}}}{m - 1} \leq \frac{N KL (Q ∣ ∣ P) + ln \frac{2 m}{δ} + 3 ln N + 1}{m - 1}

(7.3.15)

We can now choose

N

to minimize this bound.

N

may depend on

θ, Q

and the sample

S

The asymptotic bound stated in the theorem can be derived by choosing $N$ (with respect to $θ$ and $Q$ ) so as to approximately minimize the bound we have derived above. The first step in this minimization is the replacement of the Binomial tail with a looser approximation such as the Hoeffding bound which gives us: $1 - Bin (N, \delimiter N \frac{1 + θ / 2}{2}\delimiter , 0) \leq e^{- 2 N \frac{θ^{2}}{4^{2}}} = e^{- N \frac{θ^{2}}{8}}$ $Bin (N, \delimiter N \frac{1 + θ / 2}{2}\delimiter , \frac{1 + θ}{2}) \leq e^{- 2 N \frac{θ^{2}}{4^{2}}} = e^{- N \frac{θ^{2}}{8}}$

We can then choose $N = \delimiter 8 \frac{ln \frac{m}{KL (q ∣ ∣ p)}}{θ^{2}}\delimiter .$

This choice gives us:

$e^{- \frac{N θ^{2}}{8}} = \frac{KL (q ∣ ∣ p)}{m}$

Which implies we have an equation of the form:

KL (q + \frac{KL (q ∣ ∣ p)}{m} ∣ ∣ p - \frac{KL (q ∣ ∣ p)}{m}) \leq \frac{θ^{- 2} KL (q ∣ ∣ p) ln m + ln m + ln \frac{1}{δ}}{m}

(7.3.16)

In order to prove the result, we must show that:

KL (q + \frac{KL (q ∣ ∣ p)}{m} ∣ ∣ p - \frac{KL (q ∣ ∣ p)}{m}) = O (KL (q ∣ ∣ p))

(7.3.17)

We can do this by taking the difference to get an equation of the form:

$(q + k) ln \frac{q + k}{p - k} + (1 - q - k) ln \frac{1 - q - k}{1 - p + k} - q ln \frac{q}{p} - (1 - q) ln \frac{1 - q}{1 - p}$

If $p - q > 2 k$ and $k < \frac{1}{2}$ then we have:

$≃ (q + k) ln \frac{q + k + \frac{k}{p}}{p} + (1 - q - k) ln \frac{1 - q - k - \frac{k}{1 - p}}{1 - p} - q ln \frac{q}{p} - (1 - q) ln \frac{1 - q}{1 - p}$

$≃ (q + k) [ln \frac{q}{p} + \frac{k + \frac{k}{p}}{\frac{q}{p}} + (1 - q - k) ln \frac{1 - q}{1 - p} - [\frac{k + \frac{k}{1 - p}}{\frac{1 - q}{1 - p}}] - q ln \frac{q}{p} - (1 - q) ln \frac{1 - q}{1 - p}$

$≃ (q + k) [ln \frac{q}{p} + k (\frac{p + 1}{q})] + (1 - q - k) [ln \frac{1 - q}{1 - p} - k (\frac{2 - p}{1 - q})] - q ln \frac{q}{p} - (1 - q) ln \frac{1 - q}{1 - p}$

$≃ k (1 + p) - k (2 - p)$

$≃ k$

Since we have that the difference

KL (q + \frac{KL (q ∣ ∣ p)}{m} ∣ ∣ p - \frac{KL (q ∣ ∣ p)}{m}) - KL (q ∣ ∣ p) ≃ [KL (q ∣ ∣ p)]

(7.3.18)

the main theorem follows immediately.

[next] [prev] [prev-tail] [front] [up]