On Metric Entropy, Vapnik-Chervonenkis Dimension, and Learnability for a Class of Distributions

Kulkarni, Sanjeev R.

doi:10.21236/ada217331

“…Learnability for all distributions then simply imposes the uniform (upper and lower) bounds requiring the supremum over all distributions for both general (i.e., probabilistic) active learning algorithms and for deterministic algorithms. For the first part of the theorem, we need the following result relating the VC dimension of a concept class to its metric entropy: the VC dimension of C is finite iff supp N(e, C, P) < oo for all e > 0 (e.g., see [5] or [13] and references therein). The first part of the theorem follows immediately from this result.…”

Section: Theorem 2 C Is Actively Learnable For All Distributions Iff mentioning

confidence: 99%

“…The first term of the lower bound is from [13] and the second term of the lower bound follows from Lemma 2. The upper bound is from [10] which is a refinement of a result from [16] using techniques originally from [7].…”

Section: Proofmentioning

confidence: 99%

“…A considerable amount of work has been done along these lines. For example, learnability with respect to a class of distributions (as opposed to the original distribution-free framework) has been studied [5,13,14,15]. Notably, Benedek and Itai [5] first studied learnability with respect to a fixed and known probability distribution, and characterized learnability in this case in terms of the metric entropy of the concept class.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Active Learning Using Arbitrary Binary Valued Queries

KulKarni¹,

Mitter²,

Tsitsiklis³

1990

View full text Add to dashboard Cite

The original and most widely studied PAC model for learning assumes a passive learner in the sense that the learner plays no role in obtaining information about the unknown concept. That is, the samples are simply drawn independently from some probability distribution. Some work has been done on studying more powerful oracles and how they affect learnability. To find bounds on the improvement that can be expected from using oracles, we consider active learning in the sense that the learner has complete choice in the information received. Specifically, we allow the learner to ask arbitrary yes/no questions. We consider both active learning under a fixed distribution and distribution-free active learning. In the case of active learning, the underlying probability distribution is used only to measure distance between concepts. For learnability with respect to a fixed distribution, active learning does not enlarge the set of learnable concept classes, but can improve the sample complexity. For distribution-free learning, it is shown that a concept class is actively learnable iff it is finite, so that active learning is in fact less powerful than the usual passive learning model. We also consider a form of distribution-free learning in which the learner knows the distribution being used, so that 'distribution-free' refers only to the requirement that a bound on the number of queries can be obtained uniformly over all distributions. Even with the side information of the distribution being used, a concept class is actively learnable iff it has finite VC dimension, so that active learning with the side information still does not enlarge the set of learnable concept classes.*''IThis work was supported by the U.S. Army Research Office under Contract DAAL03-86-K1-0171 , by the Department. of the Na.vy under Air Force Contract F119628-9U-C-0002, and by the National Science Foundation under con t ract ECS-8552419. Report Documentation PageForm Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

show abstract

“…Comparing (8) and 7, we see that these two are of the same size when psdim(H) and BD are close to psdim(L H ) and , respectively. The leading constants in (8) are smaller by a factor of more than thirty than those in (7). As an example, when Y = 0; B] and L(z; y) = jz yj, we have that = B, D = 1, and psdim(H) psdim(L H ), som fce compares favorably with m femp .…”

Section: The Real-valued Distribution-free Casementioning

confidence: 90%

Learning by canonical smooth estimation. II. Learning and choice of model complexity

Buescher

¹

,

Kumar²

1996

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

In this paper, we analyze the properties of a procedure for learning from examples. This \canonical learner" is based on a canonical error estimator developed in a companion paper. In learning problems, we observe data that consists of labeled sample points, and the goal is to nd a model, or \hypothesis," from a set of candidates that will accurately predict the labels of new sample points. The expected mismatch between a hypothesis' prediction and the actual label of a new sample point is called the hypothesis' \generalization error." We compare the canonical learner with the traditional technique of nding hypotheses that minimize the relative frequency-based empirical error estimate. We show that, for a broad class of learning problems, the set of cases for which such empirical error minimization works is a proper subset of the cases for which the canonical learner works. We derive bounds to show that the number of samples required by these two methods is comparable. We also address the issue of how to determine the appropriate complexity for the class of candidate hypotheses. Many existing techniques solve this problem at the expense of requiring the evaluation of an absolute, a priori measure of each hypothesis' complexity. The method we present does not. It is based on an extension of the canonical learner and uses a natural, relative measure of each model's complexity. We show that this method will learn for a variety of common parametric hypothesis classes. Also, for a broad class of learning problems, we show that this method works whenever a certain conventional method for choosing model complexity does.

show abstract

“…A considerable amount of work has been done along these lines. For example, learnability with respect to a class of distributions (as opposed to the original distribution-free framework) has been studied (Benedek & Itai, 1988;Kulkarni, 1989Kulkarni, , 1991Natarajan, 1988Natarajan, , 1989. Notably, Benedek and Itai (1988) first studied learnability with respect to a fixed and known probability distribution, and characterized learnability in this case in terms of the metric entropy of the concept class.…”

Section: Introductionmentioning

confidence: 99%

Untitled

Kulkarni

¹

,

Mitter

²

,

Tsitsiklis

³

1993

Machine Learning

41

0

View full text Add to dashboard Cite

The original and most widely studied PAC model for learning assumes a passive learner in the sense that the learner plays no role in obtaining information about the unknown concept. That is, the samples are simply drawn independently from some probability distribution. Some work has been done on studying more powerful oracles and how they affect learnability. To find bounds on the improvement in sample complexity that can be expected from using oracles, we consider active learning in the sense that the learner has complete control over the information received. Specifically, we allow the learner to ask arbitrary yes/no questions. We consider both active learning under a fixed distribution and distribution-free active learning. In the case of active learning, the underlying probability distribution is used only to measure distance between concepts. For learnability with respect to a fixed distribution, active learning does not enlarge the set of learnable concept classes, but can improve the sample complexity. For distribution-free learning, it is shown that a concept class is actively learnable iff it is finite, so that active learning is in fact less powerful than the usual passive learning model. We also consider a form of distribution-free learning in which the learner knows the distribution being used, so that "distributionfree" refers only to the requirement that a bound on the number of queries can be obtained uniformly over all distributions. Even with the side information of the distribution being used, a concept class is actively learnable iff it has finite VC dimension, so that active learning with the side information still does not enlarge the set of learnable concept classes.

show abstract

On Metric Entropy, Vapnik-Chervonenkis Dimension, and Learnability for a Class of Distributions

Cited by 9 publications

References 28 publications

Active Learning Using Arbitrary Binary Valued Queries

Active Learning Using Arbitrary Binary Valued Queries

Learning by canonical smooth estimation. II. Learning and choice of model complexity

Untitled

Contact Info

Product

Resources

About