RecursiveMix: Mixed Learning with History

Abstract

Mix-based augmentation has been proven fundamental to the generalization of deep vision models. However, current augmentations only mix samples at the current data batch during training, which ignores the possible knowledge accumulated in the learning history. In this paper, we propose a recursive mixed-sample learning paradigm, termed "RecursiveMix" (RM), by exploring a novel training strategy that leverages the historical input-prediction-label triplets. More specifically, we iteratively resize the input image batch from the previous iteration and paste it into the current batch while their labels are fused proportionally to the area of the operated patches. Further, a consistency loss is introduced to align the identical image semantics across the iterations, which helps the learning of scale-invariant feature representations. Based on ResNet-50, RM largely improves classification accuracy by ∼3.2% on CIFAR100 and ∼2.8% on ImageNet with negligible extra computation/storage costs. In the downstream object detection task, the RM pretrained model outperforms the baseline by 2.1 AP points and surpasses CutMix by 1.4 AP points under the ATSS detector on COCO. In semantic segmentation, RM also surpasses the baseline and CutMix by 1.9 and 1.1 mIoU points under UperNet on ADE20K, respectively. Codes and pretrained models are available at https://github.com/megvii-research/RecursiveMix.

A Survey of Historical Learning: Learning Models with Learning History

“…Nguyen et al [137] solve the issue of network overfitting in noisy datasets by filtering out incorrect labels during training and using EMA to ensemble historical predictions. Different from above work, Yang et al [133] propose RecursiveMix, which takes the prediction from historical input as the target and aligns it with the semantics of the corresponding ROI area in the current training input. Intermediate Feature Representation.…”

“…Last, the fourth aspect focuses on employing historical input to perform data augmentation. Yang et al [133] zoom and paste the historical input onto the new round of training images to make mixed-sample data augmentation.…”

“…In contrast, checkpoints or snapshots from the learning history can be directly saved to create robust ensemble results [15]- [24]. Finally, other DEs such as prediction [133], [136], [138], gradient [12], [13],…”

Token-Label Alignment for Vision Transformers

“…Later methods including Puzzle Mix , Salien-cyMix (Uddin et al, 2020) and Attentive CutMix leverage the salient regions for informative mixture generation. Recently, Yang et al (2022) proposed a RecursiveMix strategy which employs the historical input-prediction-label triplets for scale-invariant feature learning. Despite the better performance, a drawback of these methods is the heavily increased training cost due to the saliency extraction or historical information exploitation.…”