Active Learning at the ImageNet Scale

Emam, Zeyad; Chu, Hongmin; Chiang, Ping-yeh; Czaja, Wojciech; Leapman, Richard D.; Goldblum, Micah; Goldstein, Tom

doi:10.48550/arxiv.2111.12880

Cited by 6 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ensemble active learning. Active learning iterates between training a model and selecting new inputs to be labeled [13,14,15,16,17]. In contrast, we focus on data pruning: one-shot selection of a data subset sufficient to train to high accuracy from scratch.…”

Section: Pruning Metrics: Not All Training Examples Are Created Equalmentioning

confidence: 99%

“…In order to avoid cherry-picking classes while at the same time making sure that we are visualizing images for very different classes, we here show extreme images for classes 100, ..., 500, 600 while leaving out classes 0 and 400 (which would have been part of the visualization) since those classes almost exclusively consist of images containing people (0: tench, 400: academic gown). The extremal images are shown in Figures 9,10,11,12,13,14,15,16,17,18. With larger pruning fractions, class balance decreasesboth when pruning "easy" images (turquoise) and when pruning "hard" images (orange).…”

Section: Additional Scaling Experimentsmentioning

confidence: 99%

See 1 more Smart Citation

Beyond neural scaling laws: beating power law scaling via data pruning

Sorscher¹,

Geirhos²,

Shekhar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how both in theory and practice we can break beyond power law scaling and reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this new exponential scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling performance on ResNets trained on CIFAR-10, SVHN, and ImageNet. Given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.Preprint. Under review.

show abstract

Section: Pruning Metrics: Not All Training Examples Are Created Equalmentioning

confidence: 99%

Section: Additional Scaling Experimentsmentioning

confidence: 99%

Beyond neural scaling laws: beating power law scaling via data pruning

Sorscher¹,

Geirhos²,

Shekhar³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…State-of-the-art AL algorithms have been combined with other advanced machine learning approaches to adapt the properties of specific tasks. For example, Wei et al used semisupervised learning to recognize unlabeled instances automatically [28]; Bengar et al and Emam et al integrated self-supervised learning in the AL framework [5,8]. However, the performance of the algorithm is strongly dependent on a randomly selected initial set.…”

Section: Related Workmentioning

confidence: 99%

Active Transfer Prototypical Network: An Efficient Labeling Algorithm for Time-Series Data

Yuqicheng¹,

Tnani²,

Jahnz³

et al. 2022

Preprint

View full text Add to dashboard Cite

The paucity of labeled data is a typical challenge in the automotive industry. Annotating time-series measurements requires solid domain knowledge and in-depth exploratory data analysis, which implies a high labeling effort. Conventional Active Learning (AL) addresses this issue by actively querying the most informative instances based on the estimated classification probability and retraining the model iteratively. However, the learning efficiency strongly relies on the initial model, resulting in the trade-off between the size of the initial dataset and the query number. This paper proposes a novel Few-Shot Learning (FSL)-based AL framework, which addresses the trade-off problem by incorporating a Prototypical Network (ProtoNet) in the AL iterations. The results show an improvement, on the one hand, in the robustness to the initial model and, on the other hand, in the learning efficiency of the ProtoNet through the active selection of the support set in each iteration. This framework was validated on UCI HAR/HAPT dataset and a real-world braking maneuver dataset. The learning performance significantly surpasses traditional AL algorithms on both datasets, achieving 90% classification accuracy with 10% and 5% labeling effort, respectively.

show abstract

“…Inspired by the success of deep learning in the passive regime, active learning with neural networks has been extensively explored in recent years (Sener and Savarese, 2018;Ash et al, 2019;Citovsky et al, 2021;Ash et al, 2021;Kothawade et al, 2021;Emam et al, 2021;Ren et al, 2021). Great empirical performances are observed in these papers, however, rigorous label complexity guarantees have largely remains elusive (except in Karzand and Nowak (2020); Wang et al (2021), with limitations discussed before).…”

Section: Related Workmentioning

confidence: 99%

“…When the hypothesis class is a set of neural networks, the learner further benefits from the representation power of deep neural networks, which has driven the successes of passive learning in the past decade (Krizhevsky et al, 2012;LeCun et al, 2015). With these added benefits, deep active learning has become a popular research area, with empirical successes observed in many recent papers (Sener and Savarese, 2018;Ash et al, 2019;Citovsky et al, 2021;Ash et al, 2021;Kothawade et al, 2021;Emam et al, 2021;Ren et al, 2021). However, due to the difficulty of analyzing a set of neural networks, rigorous label complexity guarantees for deep active learning have remained largely elusive.…”

Section: Introductionmentioning

confidence: 99%

Active Learning with Neural Networks: Insights from Nonparametric Statistics

Zhu¹,

Nowak²

2022

Preprint

View full text Add to dashboard Cite

Deep neural networks have great representation power, but typically require large numbers of training examples. This motivates deep active learning methods that can significantly reduce the amount of labeled training data. Empirical successes of deep active learning have been recently reported in the literature, however, rigorous label complexity guarantees of deep active learning have remained elusive. This constitutes a significant gap between theory and practice. This paper tackles this gap by providing the first near-optimal label complexity guarantees for deep active learning. The key insight is to study deep active learning from the nonparametric classification perspective. Under standard low noise conditions, we show that active learning with neural networks can provably achieve the minimax label complexity, up to disagreement coefficient and other logarithmic terms. When equipped with an abstention option, we further develop an efficient deep active learning algorithm that achieves polylog( 1 ε ) label complexity, without any low noise assumptions. We also provide extensions of our results beyond the commonly studied Sobolev/Hölder spaces and develop label complexity guarantees for learning in Radon BV 2 spaces, which have recently been proposed as natural function spaces associated with neural networks.

show abstract

Active Learning at the ImageNet Scale

Cited by 6 publications

References 0 publications

Beyond neural scaling laws: beating power law scaling via data pruning

Beyond neural scaling laws: beating power law scaling via data pruning

Active Transfer Prototypical Network: An Efficient Labeling Algorithm for Time-Series Data

Active Learning with Neural Networks: Insights from Nonparametric Statistics

Contact Info

Product

Resources

About