NoisyMix: Boosting Model Robustness to Common Corruptions

Erichson, N. Benjamin; Lim, Soon Hoe; Xu, Winnie; Utrera, Francisco; Cao, Zhenfu; Mahoney, Michael W.

doi:10.48550/arxiv.2202.01263

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A S a strong augmentation technique in supervised learning, Mixup has empirically and theoretically been proved to boost the performance of neural networks with its regularization power [2], [3], [4]. Despite its reliable performance, Mixup is also reported to strengthen deep models with better calibration [5], robustness [6], [7] and generalization [6], thus being widely used in adversarial training [4], domain adaptation [8], imbalance problems [9] and so on. However, as Mixup-style training depends heavily on data properties [10], on certain cases, chances are that traditional Mixup labels cannot correctly describe the augmented data.…”

Section: Introductionmentioning

confidence: 99%

Universum-inspired Supervised Contrastive Learning

Han¹,

Chen²

2022

Preprint

View full text Add to dashboard Cite

Mixup is an efficient data augmentation method which generates additional samples through respective convex combinations of original data points and labels. Although being theoretically dependent on data properties, Mixup is reported to perform well as a regularizer and calibrator contributing reliable robustness and generalization to neural network training. In this paper, inspired by Universum Learning which uses out-of-class samples to assist the target tasks, we investigate Mixup from a largely under-explored perspective -the potential to generate in-domain samples that belong to none of the target classes, that is, universum. We find that in the framework of supervised contrastive learning, universum-style Mixup produces surprisingly high-quality hard negatives, greatly relieving the need for a large batch size in contrastive learning. With these findings, we propose Universum-inspired Contrastive learning (UniCon), which incorporates Mixup strategy to generate universum data as g-negatives and pushes them apart from anchor samples of the target classes. Our approach not only improves Mixup with hard labels, but also innovates a novel measure to generate universum data. With a linear classifier on the learned representations, our method achieves 81.68% top-1 accuracy on CIFAR-100, surpassing the state of art by a significant margin of 5% with a much smaller batch size, typically, 256 in UniCon vs. 1024 in SupCon [1] using ResNet-50.

show abstract