3.6. Arbitrary Loss functions

A loss function is any function which takes a hypothesis, $h$, and an example $\left(x,y\right)$ as input then outputs a real number. In particular, we could choose $l\left(h,\left(x,y\right)\right)=I\left(h\left(x\right)\ne y\right)$ and regard most of the prior discussion as working with a specialized hamming loss function which is $1$ when $h\left(x\right)\ne y$ and $0$ otherwise. Many other possibilities exist. For any bounded loss function, $l\left(h,\left(x,y\right)\right)\in \left[0,1\right]$, we can define: $$e_D\left(h\right)=E_{D}l\left(h,\left(x,y\right)\right)$$ and $${\widehat{e}}_S\left(h\right)=E_{S}l\left(h,\left(x,y\right)\right)$$ All of the bounds reported here will apply for loss functions bounded on the interval $\left[0,1\right]$. The fundamental advantage that this gives us is the ability to treat the hypothesis not as a black box. Instead, we can derive a bound with a loss function dependent on the structure of the hypothesis. This will be important later when discussing averaging bounds (theorem 7.1.1) where the bound will partly depend upon the structure of the hypothesis.

For clarity of presentation, the bounds will all be presented using the hamming loss. However, they will all apply to the more general setting of arbitrary $0-1$ loss functions.