On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks

Thulasidasan, Sunil; Chennupati, Gopinath; Bilmes, Jeff; Bhattacharya, Tanmoy; Michalak, Sarah

doi:10.48550/arxiv.1905.11001

Cited by 22 publications

(34 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work inspires a range of interesting future directions, including theoretical investigations of the trade-offs between accuracy and robustness for NFM and applications of NFM beyond computer vision tasks. Further, it will be interesting to study whether NFM may also lead to better model calibration by extending the analysis of [64,81].…”

Section: Discussionmentioning

confidence: 99%

Noisy Feature Mixup

Lim¹,

Erichson²,

Utrera³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce Noisy Feature Mixup (NFM), an inexpensive yet effective method for data augmentation that combines the best of interpolation based training and noise injection schemes. Rather than training with convex combinations of pairs of examples and their labels, we use noise-perturbed convex combinations of pairs of data points in both input and feature space. This method includes mixup and manifold mixup as special cases, but it has additional advantages, including better smoothing of decision boundaries and enabling improved model robustness. We provide theory to understand this as well as the implicit regularization effects of NFM. Our theory is supported by empirical results, demonstrating the advantage of NFM, as compared to mixup and manifold mixup. We show that residual networks and vision transformers trained with NFM have favorable trade-offs between predictive accuracy on clean data and robustness with respect to various types of data perturbation across a range of computer vision benchmark datasets.

show abstract

Section: Discussionmentioning

confidence: 99%

Noisy Feature Mixup

Lim¹,

Erichson²,

Utrera³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, it was found that although modern machine learning methods, such as neural networks, made significant progress with respect to the predictive accuracy in a variety of learning tasks [30,54,55], their predictive uncertainties are poorly calibrated [24]. To mitigate this issue, several recalibration methods and their variants have been proposed, including scaling approaches [24,48,64], binning approaches [27,47,63], scaling-binning [41], and data augmentation [56,65]. Recent work has begun to develop the theoretical analysis of these methods.…”

Section: Additional Related Workmentioning

confidence: 99%

Scaffolding Sets

Burhanpurkar¹,

Deng²,

Dwork³

et al. 2021

Preprint

View full text Add to dashboard Cite

Predictors map individual instances in a population to the interval [0, 1]. For a collection C of subsets of a population, a predictor is multi-calibrated with respect to C if it is simultaneously calibrated on each set in C. We initiate the study of the construction of scaffolding sets, a small collection S of sets with the property that multi-calibration with respect to S ensures correctness, and not just calibration, of the predictor. Our approach is inspired by the folk wisdom that the intermediate layers of a neural net learn a highly structured and useful data representation.

show abstract

“…Confidence calibration: Confidence is one possible metric of inference performance, and there are a wealth of confidence calibration approaches in the literature, such as Platt/Temperature scaling [5], Histogram binning [7], Bayesian Binning into Quantiles (BBQ) [9], Isotonic regression [10], and Platt scaling extensions [8]. Several works [6], [11], [16], [17], [18], [19], [20], [21] consider either algorithmic improvements or application specific challenges associated to uncertainty quantification. [22] studies model uncertainty under dataset shift and provides empirical comparison of different calibration techniques.…”

Section: Related Workmentioning

confidence: 99%

Post-hoc Models for Performance Estimation of Machine Learning Inference

Zhang,

Oymak,

Chen

2021

Preprint

View full text Add to dashboard Cite

Estimating how well a machine learning model performs during inference is critical in a variety of scenarios (for example, to quantify uncertainty, or to choose from a library of available models). However, the standard accuracy estimate of softmax confidence is not versatile and cannot reliably predict different performance metrics (e.g., F1-score, recall) or the performance in different application scenarios or input domains. In this work, we systematically generalize performance estimation to a diverse set of metrics and scenarios and discuss generalized notions of uncertainty calibration. We propose the use of post-hoc models to accomplish this goal and investigate design parameters, including the model type, feature engineering, and performance metric, to achieve the best estimation quality. Emphasis is given to object detection problems and, unlike prior work, our approach enables the estimation of per-image metrics such as recall and F1-score. Through extensive experiments with computer vision models and datasets in three use cases -mobile edge offloading, model selection, and dataset shift -we find that proposed post-hoc models consistently outperform the standard calibrated confidence baselines. To the best of our knowledge, this is the first work to develop a unified framework to address different performance estimation problems for machine learning inference.

show abstract

On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks

Cited by 22 publications

References 0 publications

Noisy Feature Mixup

Noisy Feature Mixup

Scaffolding Sets

Post-hoc Models for Performance Estimation of Machine Learning Inference

Contact Info

Product

Resources

About