DMix: Adaptive Distance-aware Interpolative Mixup

Sawhney, Ravinder Singh; Thakkar, Megh; Pandit, Shrey; Soun, Ritesh; Jin, Dayong; Yang, Diyi; Flek, Lucie

doi:10.18653/v1/2022.acl-short.67

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, they (usually) randomly select samples to mix and do not consider the model's learning ability. Some works (Sawhney et al, 2022;Park and Caragea, 2022) have also focused on addressing this issue and proposed various methods for effectively choosing samples. Sawhney et al (2022) select samples according to the embedding similarity.…”

Section: Data Augmentation In Nlpmentioning

confidence: 99%

“…Some works (Sawhney et al, 2022;Park and Caragea, 2022) have also focused on addressing this issue and proposed various methods for effectively choosing samples. Sawhney et al (2022) select samples according to the embedding similarity. Park and Caragea (2022) merge one sample considering the confidence of the model's predictions.…”

Section: Data Augmentation In Nlpmentioning

confidence: 99%

“…However, the pseudo sample, simply combined with two samples, may not be adaptive to the model's learning ability and friendly to the model training. Recently, some work (Sawhney et al, 2022;Park and Caragea, 2022) have focused on selecting similar sample pairs for the mixup. Sawhney et al (2022) select samples according to the embedding similarity.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks

Zheng,

Zhong,

Ding

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Text classification tasks often encounter fewshot scenarios with limited labeled data, and addressing data scarcity is crucial. Data augmentation with mixup merges sample pairs to generate new pseudos, which can relieve the data deficiency issue in text classification. However, the quality of pseudo-samples generated by mixup exhibits significant variations. Most of the mixup methods fail to consider the varying degree of learning difficulty in different stages of training. And mixup generates new samples with one-hot labels, which encourages the model to produce a high prediction score for the correct class that is much larger than other classes, resulting in the model's over-confidence. In this paper, we propose a self-evolution learning (SE) based mixup approach for data augmentation in text classification, which can generate more adaptive and model-friendly pseudo samples for the model training. SE caters to the growth of the model learning ability and adapts to the ability when generating training samples. To alleviate the model over-confidence, we introduce an instance-specific label smoothing regularization approach, which linearly interpolates the model's output and one-hot labels of the original samples to generate new soft labels for label mixing up. Through experimental analysis, experiments show that our SE brings consistent and significant improvements upon different mixup methods. In-depth analyses demonstrate that SE enhances the model's generalization ability.

show abstract

Section: Data Augmentation In Nlpmentioning

confidence: 99%

Section: Data Augmentation In Nlpmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks

Zheng,

Zhong,

Ding

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…In Mixup-Transformer [16], Mixup was applied to the output of the last layer of a Transformer-based model. DMix [17] believes that fusing two samples extracted using a certain strategy will yield better results than random sampling. Therefore, they construct a set of samples whose hyperbolic distance from a sample in the dataset exceeds a set threshold, and randomly extract a sample from the set for Mixup operation.…”

Section: Interpolation-based Data Augmentationmentioning

confidence: 99%

“…For example, the 9th layer of Bert is more focused on semantic representation, while the 3rd layer is more focused on the length of the sentence. Some previous work [4,17,18] on Mixup has also studied the selection of which hidden layer output in the Bert model to use as the object of Mixup, based on the findings of literature [25].…”

Section: Selecting Hidden Layersmentioning

confidence: 99%

MSMix: An Interpolation-Based Text Data Augmentation Method Manifold Swap Mixup

Ye¹,

Wang²,

Chen³

2023

Natural Language Processing and Machine Learning

View full text Add to dashboard Cite

To solve the problem of poor performance of deep neural network models due to insufficient data, a simple yet effective interpolation-based data augmentation method is proposed: MSMix (Manifold Swap Mixup). This method feeds two different samples to the same deep neural network model, and then randomly select a specific layer and partially replace hidden features at that layer of one of the samples by the counterpart of the other. The mixed hidden features are fed to the model and go through the rest of the network. Two different selection strategies are also proposed to obtain richer hidden representation. Experiments are conducted on three Chinese intention recognition datasets, and the results show that the MSMix method achieves better results than other methods in both full-sample and small-sample configurations.

show abstract