7.4. Methods for tightening

The previous section showed a bound in asymptotic form which is good for understanding the trade-offs between the number of examples ($m$), the size of the hypothesis space ($\mid H\mid$), the margin ($\theta$) and the entropy of the average ($H\left(Q\right)$). However, it is not a good form for those interested in quantitative application of the bound to specific problems. We state improvements which aid in the development of a quantitatively applicable bound. We can tighten the bound above through several techniques:

1. Parameterizing and then optimizing the parameterization of arbitrary choices within the proof. In the (improved) margin bound proof, we arbitrarily decided to work with the margin of the randomly produced function $g\left(x\right)$ at $\frac{\theta }{2}$. This is a good heuristic, but not the optimal choice when we use the improved tail bounds. Since the decision of the margin for the random function $g\left(x\right)$ is a parameter of the proof, we are free to optimize it.
2. Tighter argument within the proof. The optimal value of $N$ is a function of $\theta ,m,\text{KL}\left(q\mid \mid p\right)$ and $\delta$. All of these are known in advance except for $\text{KL}\left(q\mid \mid p\right)$. If we can estimate in advance the value of $\text{KL}\left(q\mid \mid p\right)$, then it becomes possible to optimize the value of $N$ in a data-independent manner. Consequently, it becomes unnecessary to split our confidence over the possible values of $N$ and we need only split the confidence over the values of $\theta$ in proving the bound. The effect of this improvement is reducing $1/{\delta }_{N,k}=1/\left(N{\left(N+1\right)}^2\right)$ to $1/{\delta }_{N,k}=1/\left(N\left(N+1\right)\right)$ giving us a small improvement in the low order terms of the improved averaging bound.