CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding

Abstract

Data augmentation has been demonstrated as an effective strategy for improving model generalization and data efficiency. However, due to the discrete nature of natural language, designing label-preserving transformations for text data tends to be more challenging. In this paper, we propose a novel data augmentation framework dubbed CoDA, which synthesizes diverse and informative augmented examples by integrating multiple transformations organically. Moreover, a contrastive regularization objective is introduced to capture the global relationship among all the data samples. A momentum encoder along with a memory bank is further leveraged to better estimate the contrastive loss. To verify the effectiveness of the proposed framework, we apply CoDA to Transformer-based models on a wide range of natural language understanding tasks. On the GLUE benchmark, CoDA gives rise to an average improvement of 2.2% while applied to the Roberta-large model. More importantly, it consistently exhibits stronger results relative to several competitive data augmentation and adversarial training baselines (including the low-resource settings). Extensive experiments show that the proposed contrastive objective can be flexibly combined with various data augmentation approaches to further boost their performance, highlighting the wide applicability of the CoDA framework.

Supporting Clustering with Contrastive Learning

“…Because changing a single word can invert the meaning of the whole sentence, and the challenge is compounded when applied a second round data augmentation to short text data. However, we would hopefully cross the hurdle soon, as more effective approaches are keeping developed by the NLP community Qu et al (2020);Giorgi et al (2020); Meng et al (2021).…”

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

“…As the first step to leverage contrastive learning in sequence level pretraining, we keep everything straightforward: A simple cropping as data augmentation and the default temperature 1 is used in the softmax. Advanced data transformations (Qu et al, 2020) and hyperparameter explorations (Oord et al, 2018;Chen et al, 2020) may further improve COCO-LM but are reserved for future work.…”

Contrastive Document Representation Learning with Graph Attention Networks

“…Fang et al (2020) proposed to use back-translation to construct positive sentence pairs in their contrastive learning framework. ; Qu et al (2020) proposed multiple sentence-level augmentations strategies to do sentence contrastive learning. Most of these work still focus on either local token-level tasks or short sentence-level tasks.…”

“…Contrastive Learning offers a simple method to learn disentangled representation that encodes invariance to small and local changes in the input data without using any labeled data. In NLP domain, contrastive learning has been employed to learn sentence representation Qu et al, 2020) under either self-supervised or supervised settings.…”