TransMix: Attend to Mix for Vision Transformers

Abstract

Mixup-based augmentation has been found to be effective for generalizing models during training, especially for Vision Transformers (ViTs) since they can easily overfit. However, previous mixup-based methods have an underlying prior knowledge that the linearly interpolated ratio of targets should be kept the same as the ratio proposed in input interpolation. This may lead to a strange phenomenon that sometimes there is no valid object in the mixed image due to the random process in augmentation but there is still response in the label space. To bridge such gap between the input and label spaces, we propose TransMix, which mixes labels based on the attention maps of Vision Transformers. The confidence of the label will be larger if the corresponding input image is weighted higher by the attention map. TransMix is embarrassingly simple and can be implemented in just a few lines of code without introducing any extra parameters and FLOPs to ViT-based models. Experimental results show that our method can consistently improve various ViT-based models at scales on ImageNet classification. After pre-trained with TransMix on ImageNet, the ViT-based models also demonstrate better transferability to semantic segmentation, object detection and instance segmentation. TransMix also exhibits to be more robust when evaluating on 4 different benchmarks. Code will be made publicly available at https://github.com/Beckschen/TransMix.

RecursiveMix: Mixed Learning with History

“…Attentive CutMix [53] and SnapMix [30] take the activation map of the correct label as a confidence indicator for selecting the semantically meaningful mix regions. TransMix [5] requires the attention map of self-attention module to re-weight the targets. Manifold Mixup [52], PatchUp [17] and MoEx [36] perform a feature-level interpolation over two samples to prevent overfitting the intermediate representations.…”

Hierarchical MixUp Multi-label Classification with Imbalanced Interdisciplinary Research Proposals

“…Manifold Mixup [53], Cutmix [21], etc. ), as data augmentation methods, have not only achieved notable success in a wide range of machine learning problems such as supervised learning [8], semi-supervised learning [54,55], adversarial learning [56], but also adapted to different data forms such as images [57], texts [58,59], graphs [60], and speech [61]. Notably, to alleviate the problem of class imbalance in the dataset, a series of methods [9,10,62] employ Mixup to augment the data.…”