The Power of Comparisons for Actively Learning Linear Classifiers

Abstract

In the world of big data, large but costly to label datasets dominate many fields. Active learning, an unsupervised alternative to the standard PAC-learning model, was introduced to explore whether adaptive labeling could learn concepts with exponentially fewer labeled samples. While previous results show that active learning performs no better than its supervised alternative for important concept classes such as linear separators, we show that by adding weak distributional assumptions and allowing comparison queries, active learning requires exponentially fewer samples. Further, we show that these results hold as well for a stronger model of learning called Reliable and Probably Useful (RPU) learning. In this model, our learner is not allowed to make mistakes, but may instead answer "I don't know." While previous negative results showed this model to have intractably large sample complexity for label queries, we show that comparison queries make RPU-learning at worst logarithmically more expensive in the passive case, and quadratically more expensive in the active case.

Noise-tolerant, Reliable Active Classification with Comparison Queries

“…Our algorithms require the use of comparison queries, an addition which we show is necessary in many cases for active PAC and ARPU-learning. Along with recalling lower bounds from [6] which show comparisons are necessary for efficiently active learning non-homogeneous hyperplanes, we show that in the noiseless case it is impossible to ARPU-learn the uniform distribution over S 1 in a finite number of label queries. Further, even with the addition of a margin assumption we show the existence of simple distributions which require a number of label queries that is exponential in dimension.…”

“…By allowing the learner to ask more complicated questions of the oracle, such as comparing two points, Kane et al [4] showed that non-homogeneous linear separators in two-dimensions can be learned in exponentially fewer labeled samples than the PAC case. Later, Kane, Lovett, and Moran [5] extended this to higher dimensions using a complicated set of queries, and Hopkins, Kane, and Lovett [6] did the same by assuming weak concentration and anti-concentration on the distribution -conditions once again satisfied by s-concave distributions.…”

Active Learning Polynomial Threshold Functions

“…This encompasses not only KLMZ's [3] notion of enriched queries, but also the original "Membership query" model of Angluin [19] who allowed the learner to query any point in the overall instance space X rather than just on the subsample S ⊂ X. This model is also particularly well-studied for halfspaces where it is called the pointlocation problem [14,20,21,22,23,9], and was actually studied originally by Meyer auf der Heide [14] in the perfect learning model even before Angluin's introduction of active learning.…”

“…In this setting, the learner is given an arbitrary finite sample S ⊂ R, and must infer the labels under an adversarially chosen classifier. Variants of this model have been studied in the computational geometry [14,20,21,22,23,9], statistical learning theory [13], and clustering literatures [29,37] under various names. Formally, we say a class (X, H) is perfectly learnable with respect to a query set Q in q(n) expected queries if there exists an algorithm A such that for every n ∈ N, every sample S ⊂ X of size n, and every hypothesis h ∈ H, A correctly labels all of S with respect to h in at most q(n) queries in expectation over the internal randomness of the algorithm.…”

Efficient Learning with Arbitrary Covariate Shift

“…Some of that work, particularly the work in online algorithms with a reject option, were targeted at non-stationary sequences of examples x. Even intervals are impossible to learn in online models, and in a supervised iid model Kivinen (1990) showed that exponentially many examples are required to learning rectangles under uniform distributions (as cited by Hopkins et al (2019); Goldwasser et al (2020)). Part of the challenge is that most definitions also require few test rejections, unlike PQ-learning's requirement of few rejections with respect to P .…”