Our real goal in learning theory is to answer the question “When can we learn?” Unfortunately, there is no good answer to this question given only the assumption of independence. In particular, it may be impossible to learn. The simplest example of such a learning problem is the case of a distribution which always flips a coin in deciding the value of the output . The bias of the coin is the same irrespective of the input . Since the value of the input is explicitly independent of the output we surely can not hope to learn a useful relation between the input and output.
Considerable work has been done elsewhere to answer “When can we learn?” Typically this is done in models with stronger assumptions—for example, under the additional assumption that the output is related to the input by an “OR” of a subset of the input bits.
We will instead focus on a different question: “Have we learned?” This question is answerable in a probabilistic manner. In particular we can make a statement such as “With high probability over samples drawn from we have learned if the empirical error is less than some value.” In practice, we will want to know how much we have learned which we can do by providing a high confidence bound on the true error rate of the learned hypothesis.