Learning by canonical smooth estimation. II. Learning and choice of model complexity

Abstract

In this paper, we analyze the properties of a procedure for learning from examples. This \canonical learner" is based on a canonical error estimator developed in a companion paper. In learning problems, we observe data that consists of labeled sample points, and the goal is to nd a model, or \hypothesis," from a set of candidates that will accurately predict the labels of new sample points. The expected mismatch between a hypothesis' prediction and the actual label of a new sample point is called the hypothesis' \generalization error." We compare the canonical learner with the traditional technique of nding hypotheses that minimize the relative frequency-based empirical error estimate. We show that, for a broad class of learning problems, the set of cases for which such empirical error minimization works is a proper subset of the cases for which the canonical learner works. We derive bounds to show that the number of samples required by these two methods is comparable. We also address the issue of how to determine the appropriate complexity for the class of candidate hypotheses. Many existing techniques solve this problem at the expense of requiring the evaluation of an absolute, a priori measure of each hypothesis' complexity. The method we present does not. It is based on an extension of the canonical learner and uses a natural, relative measure of each model's complexity. We show that this method will learn for a variety of common parametric hypothesis classes. Also, for a broad class of learning problems, we show that this method works whenever a certain conventional method for choosing model complexity does.

Model selection by bootstrap penalization for classification

“…In this paper, we focus on random penalty functions. This subject has been tackled by Buescher and Kumar (1996), Lugosi and Nobel (1999) and Boucheron et al (2000) but the most interesting works for our approach are the ones due to Koltchinskii (2001) and Bartlett et al (2002). Let ξ denote the sample (X 1 , Y 1 ), .…”

“…This remark naturally leads to the idea of data-driven penalization. Buescher and Kumar (1996), Lugosi and Nobel (1999), Boucheron et al (2000) introduce some penalties involving the related empirical coverings or empirical Shatter coefficients. Inspired by the method of Rademacher symmetrization commonly used in the empirical processes theory (see for instance Van der Vaart and Wellner, 1996), Koltchinskii (2001) and Bartlett et al (2002) independently propose the so-called Rademacher penalties which are based on random variables of the form:…”

A Data-Dependent Skeleton Estimate and a Scale-Sensitive Dimension for Classification

“…Since jXj = m, there exists an x 2 X such that this property holds for at least 72k pairs. For j 2 f 1 : : : b g, de ne By the pigeonhole principle, there are at least 72k= 9 2 = 2 k pairs (h 1 h 2 ) for which the set f (h 1 (x) h 2 (x))g is the same. Then it follows that there are two subclasses H 1 H 2 H and indeces i j 2 f 1 : : : 9g with jH 1 j = jH 2 j = 2 k such that for each h 1 2 H 1 , (h 1 (x)) = i, f o r each h 2 2 H 2 , (h 2 (x)) = j, a n d i j + 4.…”