3.5. Bounding the Union

One very common technique we will use is the union bound (known as the Bonferroni bound in statistics). Given two coins, each with a bias (probability of heads) of $p$, what is the probability that if we flip each coin $m$ times, one of the coins will have $k$ or fewer heads?

Let $X_1$ = the proportion of heads in the first coin flip and $X_2$ = the proportion of heads in the second coin flip. Then we get: $$Pr\left(X_1\le \frac{k}{m}\text{ or }{X}_2\le \frac{k}{m}\right)$$ $$\le Pr\left(X_1\le \frac{k}{m}\right)+Pr\left(X_2\le \frac{k}{m}\right)$$ where the inequality is known as the union bound. This step is very applicable because it works even when the values of $X_1$ and $X_2$ are correlated in arbitrary ways.

The union bound is the fundamental tool which allows us to reason about multiple hypotheses, each with possibly correlated empirical errors, $X_1={\widehat{e}}_S\left(h_1\right)$ and $X_2={\widehat{e}}_S\left(h_2\right)$. The use (and avoidance of the use) of the union bound is a constant issue.