Accelerating Deep Learning by Focusing on the Biggest Losers

Jiang, Angela H.; Wong, Daniel L. -K.; Zhou, Giulio; Andersen, David G.; Dean, Jeffrey; Ganger, Gregory R.; Joshi, Gauri; Kaminksy, Michael; Kozuch, Michael A.; Pillai, Padmanabhan

doi:10.48550/arxiv.1910.00762

Cited by 10 publications

(20 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With 1% labeled data on the CIFAR-10 dataset, the proposed framework achieves 28.36% higher accuracy than using the 1% labeled data for direct supervised learning. The proposed contrast scoring based data selection achieves 13.9% higher accuracy than the SOTA data selection approach [13]. Meanwhile, the proposed approach achieves 2.67x faster learning than the baseline when the same accuracy is achieved.…”

Section: Introductionmentioning

confidence: 88%

See 1 more Smart Citation

Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast

Wu¹,

Wang²,

Zeng³

et al. 2021

Preprint

View full text Add to dashboard Cite

After a model is deployed on edge devices, it is desirable for these devices to learn from unlabeled data to continuously improve accuracy. Contrastive learning has demonstrated its great potential in learning from unlabeled data. However, the online input data are usually none independent and identically distributed (non-iid) and edge devices' storages are usually too limited to store enough representative data from different data classes. We propose a framework to automatically select the most representative data from the unlabeled input stream, which only requires a small data buffer for dynamic learning. Experiments show that accuracy and learning speed are greatly improved.

show abstract

Section: Introductionmentioning

confidence: 88%

“…The next two baselines are SOTA approaches to select data for efficient training and improving accuracy. Selective-Backprop [13] selects data with the largest losses for training. K-Center is a SOTA active learning approach [25], which selects the most representative data by performing k-center clustering in the features space.…”

Section: A Experimental Setupmentioning

confidence: 99%

Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast

Wu¹,

Wang²,

Zeng³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A body of recent work has emphasized making more computationally efficient models (Wu et al, 2019;Coleman et al, 2019;Jiang et al, 2019), yet another line of work has focused on the opposite: building larger models with more parameters to tackle more complex tasks (Amodei and Hernandez, 2018;Sutton, 2019). We suggest leaderboards which utilize carbon emissions and energy metrics to promote an informed balance of performance and efficiency.…”

Section: Energy Efficiency Leaderboardsmentioning

confidence: 99%

Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning

Henderson¹,

Hu²,

Romoff³

et al. 2020

Preprint

View full text Add to dashboard Cite

Accurate reporting of energy and carbon usage is essential for understanding the potential climate impacts of machine learning research. We introduce a framework that makes this easier by providing a simple interface for tracking realtime energy consumption and carbon emissions, as well as generating standardized online appendices. Utilizing this framework, we create a leaderboard for energy efficient reinforcement learning algorithms to incentivize responsible research in this area as an example for other areas of machine learning. Finally, based on case studies using our framework, we propose strategies for mitigation of carbon emissions and reduction of energy consumption. By making accounting easier, we hope to further the sustainable development of machine learning experiments and spur more research into energy efficient algorithms.

show abstract

“…Alternatively one can re-weight examples according to their loss function when using a stochastic optimizer, which can be used to put more emphasis on "hard" examples [24,29,57]. Re-weighting can also be implicitly enforced via the inclusion of a regularization parameter [1], loss clipping [69], or modelling of crowd-worker qualities [30], which can make the objective more robust to rare instances.…”

Section: Related Workmentioning

confidence: 99%

Tilted Empirical Risk Minimization

Li,

Beirami,

Sanjabi

et al. 2020

Preprint

View full text Add to dashboard Cite

Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly. While many methods aim to address these problems individually, in this work, we explore them through a unified framework-tilted empirical risk minimization (TERM). In particular, we show that it is possible to flexibly tune the impact of individual losses through a straightforward extension to ERM using a hyperparameter called the tilt. We provide several interpretations of the resulting framework: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to a superquantile method. We develop batch and stochastic first-order optimization methods for solving TERM, and show that the problem can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. TERM is not only competitive with existing solutions tailored to these individual problems, but can also enable entirely new applications, such as simultaneously addressing outliers and promoting fairness.

show abstract

Accelerating Deep Learning by Focusing on the Biggest Losers

Cited by 10 publications

References 0 publications

Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast

Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrast

Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning

Tilted Empirical Risk Minimization

Contact Info

Product

Resources

About