Progressive Validation provides an improvement over the simple technique of using a holdout set by allowing almost half (on average) of the test set examples to be used in training without loosening the resulting bound when we compare to the Hoeffding bound for a holdout set. In practice, limiting ourselves to using the Hoeffding bound for holdout sets is not acceptable so progressive validation may not (in practice) have a better bound than the holdout bound. For this reason, progressive validation is not compared with other bound calculations in later experiments.
Tightening the analysis to statements about Binomial tail bounds is a significant open problem.