2.1 Formal Model

2.1. Formal Model

Let $X$ be the space of the input to a predictor and $Y$ be the space of the output. A labeled example $(x, y)$ consists of an input, $x$ and the desired output, $y$ . Our formal model starts with the assumption that all labeled examples are drawn independently from a distribution $D$ over the space $X \times Y$ . This is strictly more general than the ’target concept’ model which assumes that there exists some function $f : X \to Y$ used to generate the label [50]. In particular we can model probabilistic learning problems which do not have a particular $Y$ value for each $X$ value. This generalization is essentially “free” in the sense that it does not add to the complexity of presenting the results.

The set of $m$ independently drawn samples presented to a learning algorithm will be denoted as $S$ . The learning algorithm will output a hypothesis $h : X \to Y$ which has some unobservable true error rate $e_{D} (h)$ and an observable empirical error rate, ${\hat{e}}_{S} (h)$ .

DEFINITION 2.1.1.

(True error) The true error $e_{D} (h)$ of a hypothesis $h$ is defined in the following way: $e_{D} (h) = {Pr}_{D} (h (x) \neq y)$

Unfortunately, the true error is not an observable quantity in our model because the distribution, $D$ , is unknown. However, there is a related quantity which is observable.

DEFINITION 2.1.2.

(Empirical Error) Given a sample set $S$ , the empirical error, ${\hat{e}}_{S} (h)$ is defined as: ${\hat{e}}_{S} (h) = {Pr}_{S} (h (x) \neq y) = \frac{1}{m} \sum_{i = 1}^{m} I (h (x_{i}) \neq y_{i})$ where $I ()$ is a function which maps “true” to $1$ and “false” to $0$ . Here ${Pr}_{S} (. . .)$ is a probability taken with respect to the uniform distribution over the set of examples, $S$ .

[next] [front] [up]