9.2. The Setting and Prior Results

We will first discuss standard covering number bounds. Define a “distance” in terms of how often hypotheses disagree according to: $$d_D\left(h,f\right)={Pr}_D\left(h\left(x\right)\ne f\left(x\right)\right)$$Now, start with an epsilon net defined by: $$N\left(H,\epsilon ,d_D\right)={\text{inf}}_F\left|F:\forall h\in H\exists f\in F:d_D\left(h,f\right)\le \epsilon \right|$$ An epsilon net is the minimum size of a set which contains an element “near” to every element in $H$.

Then a covering number is defined as: $$C\left(H,\epsilon \right)={\text{sup}}_{D}N\left(H,\epsilon ,d_D\right)$$ The covering number is the worst epsilon net.

How tight is this bound when applied to a finite independent hypothesis space? We can improve the constants by using an argument with fewer triangle inequalities in the discrete case and get the following results: $${Pr}_D\left(e\left(h\right)-\widehat{e}\left(h\right)\ge 2\sqrt{\frac{ln4\mid H\mid +ln\delta }{m}}\right)\le \delta$$ Comparing this with a very loose application of the discrete hypothesis bound 4.2.3 we see that the penalty term in the covering number bound is worse by factor of $2\sqrt{2}$. Put another way, dividing the number of samples by $8$or increasing the hypothesis space size to $\mid H{\mid }^8$ and then applying a sloppy discrete hypothesis bound is about equivalent to applying a very specialized covering number bound. We seek a covering number bound which does not divide the effective value of a hypothesis by $8$.