Fat-Shattering and the Learnability of Real-Valued Functions

Abstract

We consider the problem of learning real-valued functions from random examples when the function values are corrupted with noise. With mild conditions on independent observation noise, we provide characterizations of the learnability of a real-valued function class in terms of a generalization of the Vapnik Chervonenkis dimension, the fat-shattering function, introduced by Kearns and Schapire. We show that, given some restrictions on the noise, a function class is learnable in our model if an only if its fat-shattering function is finite. With different (also quite mild) restrictions, satisfied for example by guassion noise, we show that a function class is learnable from polynomially many examples if and only if its fat-shattering function grows polynomially. We prove analogous results in an agnostic setting, where there is no assumption of an underlying function class. ] 1996 Academic Press, Inc.

Valid Generalisation from Approximate Interpolation

“…This means that the band-dimension is not simply one number depending on H, but is, rather, a function depending on H. (A number of such scale-sensitive dimensions have proven to be useful in learning theory [16,1,9,23,24].) Let H be a set of real-valued functions.…”

“…This dimension was introduced by Kearns and Schapire [16] in their work on the learnability of p-concepts. Here, we use the notation and terminology of [9]. Suppose that H is a set of functions from X to [0, 1] and that γ > 0.…”

“…In particular, the pac learning of {0, 1}-valued functions (equivalently, sets) has been studied in great depth; see [12,5,18], for example. More recently, attention has been focussed on the extension of the pac model to classes of real-valued functions; see, for example, [14,1,9]. The problem studied in this paper is a problem in probability theory which is motivated by, and has applications to, the learnability of real-valued function classes.…”

Untitled

“…However, in all these cases one obtains covering number bounds which behave like O( −D ) for some generalized dimension D. If this is the case, replace the pseudo-dimension by the dimension D, and all the results below follow. Note, however, that in some cases D may depend on , and a more careful analysis is needed using the so-called fat-shattering dimension as is discussed in Bartlett et al (1996). Furthermore, there are specific situations where the pseudo-dimension yields nearly optimal bounds for the estimation error, since in that case the pseudo-dimension is essentially equivalent to another combinatorial dimension, called the fat-shattering dimension, which gives nearly matching lower bounds on estimation error (see Bartlett et al (1996)).…”

Untitled

“…We begin with the definition of the fat shattering dimension, which was first introduced in Kearns and Schapire (1990), and has been used for several problems in learning since (Alon et al, 1997;Bartlett, Long, & Williamson, 1996;Anthony & Bartlett, 1994;Bartlett & Long, 1995).…”