Universum-inspired Supervised Contrastive Learning

Abstract

Mixup is an efficient data augmentation method which generates additional samples through respective convex combinations of original data points and labels. Although being theoretically dependent on data properties, Mixup is reported to perform well as a regularizer and calibrator contributing reliable robustness and generalization to neural network training. In this paper, inspired by Universum Learning which uses out-of-class samples to assist the target tasks, we investigate Mixup from a largely under-explored perspective -the potential to generate in-domain samples that belong to none of the target classes, that is, universum. We find that in the framework of supervised contrastive learning, universum-style Mixup produces surprisingly high-quality hard negatives, greatly relieving the need for a large batch size in contrastive learning. With these findings, we propose Universum-inspired Contrastive learning (UniCon), which incorporates Mixup strategy to generate universum data as g-negatives and pushes them apart from anchor samples of the target classes. Our approach not only improves Mixup with hard labels, but also innovates a novel measure to generate universum data. With a linear classifier on the learned representations, our method achieves 81.68% top-1 accuracy on CIFAR-100, surpassing the state of art by a significant margin of 5% with a much smaller batch size, typically, 256 in UniCon vs. 1024 in SupCon [1] using ResNet-50.

Class-Aware Universum Inspired Re-Balance Learning for Long-Tailed Recognition

“…Chidambaram et al [25] analyzed that the advantages of mixup depend on the properties of the data. Han et al [48] analyzed that the failure of the mixup may be that the synthesized data points are still soft-connected to the original labels, and propose a Universum-style mixup that is disconnected from all known class labels.…”

“…To avoid the effect of label combination, Han et al [48] used the images generated by only using sample-level mixup operation as Universum for supervised contrastive learning, and the model performance was significantly improved. However, this operation may violate the definition of Universum.…”