3.4 Converting to a P-value approach

3.4. Converting to a P-value approach

When making judgments about which hypothesis to choose, the relevant quantity is not the probability of error as we calculate above. Instead, it is a bound on the true error rate which holds with high probability over draws of the sample set. We might decide that $δ = 0.05$ was an acceptable rate of bound failure and then ask ourselves, “What is a bound on the true error rate that holds with probability $0.9$ ?”

Functionally, instead of calculating: $Bin (m, k, p) = δ$ we want to invert the output, $δ$ , with respect to the input, $p$ . Since $p$ and $δ$ are monotonically related to each other, this inversion can be defined as: $\bar{e} (m, k, δ) \equiv {max}_{p} \{p : Bin (m, k, p) = δ\}$

What is the interpretation of $\bar{e}$ ? The inversion $\bar{e} (m, k, δ)$ is a high confidence bound on the true error rate. With probability at least $1 - δ$ (over the draws of the examples), the true error rate will be less than $\bar{e} (m, k, δ)$ . This is exactly the kind of quantity that we desire in making decisions about which hypothesis is more desirable. This calculation has been done for several values of $m$ and Binomial tail bounds in figure ( 3.4.1).

We will use the process of inversion in many places. The fundamental soundness of inversion rests upon the following lemma.

LEMMA 3.4.1.

(Inversion Lemma) For all predicates, $φ (X)$ ${sup}_{P} {Pr}_{X \sim P} (φ_{P} (X)) \leq δ \Rightarrow \forall P {Pr}_{X \sim P} (φ_{P} (X)) \leq δ$

This lemma is the trivial statement that a set of objects is less than the $sup$ over the set of objects. Nonetheless, it is a very important step which will appeal to implicitly and explicitly later on.

PROOF. By contradiction. Assume there exists $P$ such that the right hand side is not satisfied. Then, the left hand side can not be correct. ▫

The Inversion lemma allows us to implicitly parameterize all precision parameters $ε (δ)$ in terms of a fixed probability of failure, $δ$ . This is important for practical application because it means we can choose our probability of failure $δ$ before looking at any examples.

Figure 3.4.1:

A plot of the difference between the true error upper bound and the empirical error for various Binomial tail bound approximations. Here

m = 1000

coins of an unknown bias are used and a confidence of

δ = 0.1

and the horizontal axis is the number of errors between

k = 0

and

k = 1000

. The relative entropy approximation to the Binomial tail is relatively well behaved while the Hoeffding approximation is not. In particular, the Hoeffding approximation does not take into account the decreased variance of low bias Binomials. Note that the dip at the end of the Hoeffding bound is due to the fact that the true error rate is always less than

1

[next] [prev] [prev-tail] [front] [up]