A Data-Dependent Skeleton Estimate and a Scale-Sensitive Dimension for Classification

Abstract

The classical binary classi cation problem is investigated when it is known in advance that the posterior probability function (or regression function) belongs to some class of functions. We i n troduce and analyze a method which e ectively exploits this knowledge. The method is based on minimizing the empirical risk over a carefully selected \skeleton" of the class of regression functions. The skeleton is a covering of the class based on a data-dependent metric, especially tted for classi cation. A new scale-sensitive dimension is introduced which is more useful for the studied classi cation problem than other, previously de ned, dimension measures. This fact is demonstrated by performance bounds for the skeleton estimate in terms of the new dimension. 1

Learning pattern classification-a survey

“…Covering numbers enter naturally in learning problems when one replaces an infinite class of rules by a carefully selected finite subset of rules, and then attempts to use empirical risk minimization (or other learning rule) over this finite subcollection. Work along these lines can be found in [40], [53], [102], [155], [175], [274], and [285].…”