1.4. The oblivious passive supervised learning model

Oblivious will be modeled by an unknown distribution $D$ over examples. Here, an “example” is just a vector of observations. Since this is a supervised learning model, all of our examples will split into two parts, $\left(x,y\right)$ where $x$ is the “input” and $y$ is the “output” (the thing we wish to predict). A quick example is predicting whether precipitation will be in the form of rain or snow (“$y$” value) given the temperature (“$x$” value).

For simplicity, we will typically work with theorems for binary valued $y$. We can remove this choice by generalizing sample complexity bounds—but we do not do so for simplicity of presentation.

The fundamental assumption we will make in all of our sample complexity bounds is that all examples are drawn independently from the unknown distribution $D$. This assumption must be stated explicitly and always kept in mind when considering the relevance of sample complexity bounds.

With the exception of this assumption, all of the other parameters in our bounds will be verifiable at the time the bound is applied.

Note that we use a distribution over labeled examples and not a combination of a distribution over the input space along with a function from the input space to the output space as in many other formulations. This choice is made because it is both more general and mathematically simpler.

The number of samples, $m$, required for learning is the fundamental quantity we will be concerned with. In particular, we will not be concerned with the time complexity or the space complexity of learning algorithms. This choice is made for the purposes of simplicity and implies that the relationship between sample complexity bounds and learning algorithms will be similar to the difference between information theory and coding information for transmission across a noisy channel.

Any learning algorithm must output some hypothesis, $h$, for predicting the output given the input. This hypothesis is essentially a program which, given the input, predicts the output. The hypothesis may or may not be randomized—it might choose an output deterministically or according to some randomization.

The next item to quantify is learning. When has learning occurred? We will say that learning has occurred when the true error is significantly less than a uniform random prediction. The true error $e_D\left(h\right)$ is defined in the following way: $$e_D\left(h\right)={Pr}_D\left(h\left(x\right)\ne y\right)$$

Unfortunately, the true error is not an observable quantity in our model because the distribution, $D$, is unknown. However, there is a related quantity which is observable. Given a sample set $S$ of $\left(x,y\right)$ pairs $\left\{\left(x_1,y_1\right),...,\left(x_m,y_m\right)\right\}$, the empirical error, ${\widehat{e}}_S\left(h\right)$ is defined similarly as: $${\widehat{e}}_S\left(h\right)={Pr}_S\left(h\left(x\right)\ne y\right)=\frac{1}{m}{\sum }_{i=1}^{m}I\left(h\left(x_i\right)\ne y_i\right)$$ where $I\left(\right)$ is a function which maps “true” to $1$ and “false” to $0$. Here ${Pr}_S\left(...\right)$ is a probability taken with respect to the uniform distribution over the set of examples, $S$.