2.2. Relationship to Prior Work

2.2.1. Distribution free learning

The question answered here differs significantly from much prior learning theory, including the results of Vapnik [51], Valiant [50], Devroye [11], and many others. See [20] for a good summary. The principle difference is the question we address: “Have we learned?”

Much of the prior work in learning theory addresses the question: “How many examples are needed in order to guarantee that I will choose (nearly) the best hypothesis from some fixed hypothesis set?”

In particular, suppose that we have a hypothesis set $H$. If we guarantee that:

$${Pr}_{S\sim D^m}\left(\exists h\in H:\mid e_D\left(h\right)-{\widehat{e}}_S\left(h\right)\mid >\epsilon \right)<\delta$$ then if our learning algorithm choses the hypothesis with minimum empirical error (known as “empirical risk minimization” in the language of [52]), we will be guaranteed that the chosen hypothesis satisfies: $$e_D\left(h\right)-e_D^{\ast }\le 2\epsilon$$ where $e_D^{\ast }={inf}_{h\in H}{e}_D\left(h\right)$.

There are several difficulties inherent in this approach which we will avoid.

1. Results in this model apply only for the empirical risk minimization (ERM) algorithm. The ERM algorithm is known to be NP-complete [4] for some hypothesis spaces and, in general, is essentially dependent upon the axiom of choice. These results will approximately apply to approximate ERM algorithms, but it is unclear how “approximate” typical learning algorithms are. By answering “Have we learned?” this complexity is avoided.
2. There is no natural notion of preference (or “prior”) amongst the hypotheses, $h$, in the hypothesis space, $H$. This is very important for practical application as is shown in Figure 12.3.3.
3. Answers to this question are generally insensitive to the final result, ${\widehat{e}}_S\left(h\right)$. This is again important in practice (shown here 3.4.1) because the variance of the distribution of the empirical error, ${\widehat{e}}_S\left(h\right)$, changes significantly with different true error rates, $e_D\left(h\right)$.

These drawbacks can be alleviated (but not removed) to some extent. For example, many people apply results in this model to arbitrary learning algorithms by simply noticing that the deviation of the empirical and true error rates is small. This, in turn, implies that whatever hypothesis your algorithm learns (empirical minimum or not), it’s true error is within $\epsilon$ of the empirical error. This is one approach to addressing the question “have we learned?” - others will be presented here.

The second drawback can be alleviated using “structural risk minimization” (as in [52]). Structural risk minimization removes most of drawback (2), although it is awkward for specifying very fine-grained preferences. We will use an arbitrary measure $P$ over the hypothesis space $h$. This prior $P$ need not (necessarily) be a Bayesian prior - all that is formally required is that this “prior” be specified without using information from the examples. The notion of measure $P$ is more general than structural risk minimization because for every “structure” $T$ on which structural risk minimization is done, we can produce a “prior” $P_T$ dependent upon the structure $T$ and derive the same (or tighter) results.

The third drawback can be alleviated using “relative risk” as in [52], but this still leaves some slack in the bounds.

The work in this thesis can be thought of as directly addressing the altered question which alleviates problem 1. Since our goal is addressing this altered question rather than deriving the answer from other results, we will be able to state and prove tighter results. Furthermore, because we address the question people encounter in practice, the bounds presented here will be more directly applicable. In particular, they will apply to arbitrary learning algorithms (although not necessarily tightly to arbitrary learning algorithms) rather than just the empirical risk minimization algorithm.

2.2.2. Bayesian Analysis

The basic result of Bayesian analysis is that Bayes rule: $$Pr\left(f\mid S\right)=\frac{Pr\left(S\mid f\right)Pr\left(f\right)}{Pr\left(S\right)}$$ is the optimal learning algorithm when the learning problem is drawn from the distribution $Pr\left(f\right)$. Note that the “hypothesis” learned by the Bayesian learning algorithm is the weighted average predictor, $$h\left(x\right)=\text{sign}\left(\int Pr\left(f\mid S\right)f\left(x\right)df\right)$$ This rather strong statement is difficult to utilize in practice because specifying and using arbitrary distributions $Pr\left(f\right)$ is unwieldy. In practice, many people use approximations and there is considerable question about whether or not the specified $Pr\left(f\right)$ is “right enough” after approximations. A chapter is later devoted to analyzing hypotheses of this weighted-average form and a theorem 7.2.1 about their accuracy is proved.

Some work has been done to analyze the robustness of Bayesian algorithms under approximation errors. There are two common traits of Bayesian-related analysis:

1. All statements are parameterized by a prior, $P$.
2. The analysis is typically an “average case” (w.r.t. the prior $P$ and a fixed Bayesian or approximate Bayesian algorithm, $A$).

An example of this sort of analysis can be found in [21]. The work in this thesis adopts parameterization by a measure $P$, but is not an average case analysis. Our analysis is “worst-case” in the sense that it applies to all learning problems whether or not the measure $P$ is a “correct” prior or not. This approach is strongly similar to the work of McAllester [39], and a later chapter is devoted to a refinement of this result. Despite this, it is interesting to note that our bounds are minimized (in some sense) when the measure $P$ is in fact a “correct” Bayesian prior.

There have been other attempts to connect Bayesian average case settings with worst case settings. One interesting example is [16] which discusses the connection between Bayesian setting and the mistake bound model. This is especially interesting because the mistake bound model is, in some sense, more “worst-case” than the model we consider here as no assumption of example independence is made. Further work [29] has occurred in the “Minimum Description Length” (MDL) community.