Active Learning for Classification With Abstention

Shekhar, Shubhanshu; Ghavamzadeh, Mohammad; Javidi, Tara

doi:10.1109/jsait.2021.3081433

Cited by 5 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Exact learning might be possible with equivalence queries [51] in addition to the membership queries to the (human) teacher. As a more general approach, we also plan to evaluate binary classification with abstention [52]. Only test inputs with "uncertain" label are passed to the human.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Human-In-The-Loop Automatic Program Repair

Böhme¹,

Geethal²,

Pham³

2019

Preprint

View full text Add to dashboard Cite

We introduce LEARN2FIX, the first human-in-theloop, semi-automatic repair technique when no bug oracle-except for the user who is reporting the bug-is available. Our approach negotiates with the user the condition under which the bug is observed. Only when a budget of queries to the user is exhausted, it attempts to repair the bug. A query can be thought of as the following question: "When executing this alternative test input, the program produces the following output; is the bug observed"? Through systematic queries, LEARN2FIX trains an automatic bug oracle that becomes increasingly more accurate in predicting the user's response. Our key challenge is to maximize the oracle's accuracy in predicting which tests are bug-revealing given a small budget of queries. From the alternative tests that were labeled by the user, test-driven automatic repair produces the patch.Our experiments demonstrate that LEARN2FIX learns a sufficiently accurate automatic oracle with a reasonably low labeling effort (lt. 20 queries). Given LEARN2FIX's test suite, the GenProg test-driven repair tool produces a higher-quality patch (i.e., passing a larger proportion of validation tests) than using manual test suites provided with the repair benchmark.1 We present a randomized variant of the algorithm presented by Holub et al. [10]. We do not need to construct two classifiers for each unlabeled data point in a fixed-size sample. Instead, our sample is continuously generated.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Human-In-The-Loop Automatic Program Repair

Böhme¹,

Geethal²,

Pham³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The unit ball in the Sobolev space is defined as W α,∞ 1 (X ) := {f : f W α,∞ ≤ 1}. Following the convention of nonparametric active learning (Castro and Nowak, 2008;Minsker, 2012;Locatelli et al, 2017Locatelli et al, , 2018Shekhar et al, 2021;Kpotufe et al, 2021), we assume X = [0, 1] d and η ∈ W α,∞ 1 (X ) (except in Section 4).…”

Section: Problem Settingmentioning

confidence: 99%

“…In particular, Puchkin and Zhivotovskiy (2021); Zhu and Nowak (2022) develop active learning algorithms with polylog( 1 ε ) label complexity when analyzed under Chow's excess error. Shekhar et al (2021) study nonparametric active learning under a different notion of the Chow's excess error, and propose algorithms with poly( 1 ε ) label complexity; their algorithms follow similar procedures of those partition-based nonparametric active learning algorithms (e.g., Minsker (2012); Locatelli et al (2017)).…”

Section: Related Workmentioning

confidence: 99%

“…We obtain insights from the nonparametric setting where the conditional probability (of taking a positive label) is assumed to be a smooth function (Tsybakov, 2004;Audibert and Tsybakov, 2007). Previous nonparametric active learning algorithms proceed by partitioning the action space into exponentially many sub-regions (e.g., partitioning the unit cube [0, 1] d into ε −d sub-cubes each with volume ε d ), and then conducting local mean (or some higher-order statistics) estimation within each sub-region (Castro and Nowak, 2008;Minsker, 2012;Locatelli et al, 2017Locatelli et al, , 2018Shekhar et al, 2021;Kpotufe et al, 2021). We show that, with an appropriately chosen set of neural networks that globally approximates the smooth regression function, one can in fact recover the minimax label complexity for active learning, up to disagreement coefficient (Hanneke, 2007(Hanneke, , 2014 and other logarithmic factors.…”

Section: Introductionmentioning

confidence: 99%

“…As a byproduct, our results also provide a new perspective on nonparametric active learning through the lens of neural network approximations. Nonparametric active learning was previously tackled through space partitioning and local estimations over exponentially many sub-regions(Castro and Nowak, 2008;Minsker, 2012;Locatelli et al, 2017Locatelli et al, , 2018Shekhar et al, 2021;Kpotufe et al, 2021).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Active Learning with Neural Networks: Insights from Nonparametric Statistics

Zhu¹,

Nowak²

2022

Preprint

View full text Add to dashboard Cite

Deep neural networks have great representation power, but typically require large numbers of training examples. This motivates deep active learning methods that can significantly reduce the amount of labeled training data. Empirical successes of deep active learning have been recently reported in the literature, however, rigorous label complexity guarantees of deep active learning have remained elusive. This constitutes a significant gap between theory and practice. This paper tackles this gap by providing the first near-optimal label complexity guarantees for deep active learning. The key insight is to study deep active learning from the nonparametric classification perspective. Under standard low noise conditions, we show that active learning with neural networks can provably achieve the minimax label complexity, up to disagreement coefficient and other logarithmic terms. When equipped with an abstention option, we further develop an efficient deep active learning algorithm that achieves polylog( 1 ε ) label complexity, without any low noise assumptions. We also provide extensions of our results beyond the commonly studied Sobolev/Hölder spaces and develop label complexity guarantees for learning in Radon BV 2 spaces, which have recently been proposed as natural function spaces associated with neural networks.

show abstract

Active learning algorithm through the lens of rejection arguments

Denis,

Hebiri,

Ndjia Njike

et al. 2023

Mach Learn

View full text Add to dashboard Cite

Active learning is a paradigm of machine learning which aims at reducing the amount of labeled data needed to train a classifier. Its overall principle is to sequentially select the most informative data points, which amounts to determining the uncertainty of regions of the input space. The main challenge lies in building a procedure that is computationally efficient and that offers appealing theoretical properties; most of the current methods satisfy only one or the other. In this paper, we use the classification with rejection in a novel way to estimate the uncertain regions. We provide an active learning algorithm and prove its theoretical benefits under classical assumptions. In addition to the theoretical results, numerical experiments are carried out on synthetic and non-synthetic datasets. These experiments provide empirical evidence that the use of rejection arguments in our active learning algorithm is beneficial and allows good performance in various statistical situations.

show abstract

Active Learning for Classification With Abstention

Cited by 5 publications

References 20 publications

Human-In-The-Loop Automatic Program Repair

Human-In-The-Loop Automatic Program Repair

Active Learning with Neural Networks: Insights from Nonparametric Statistics

Active learning algorithm through the lens of rejection arguments

Contact Info

Product

Resources

About