Generative Data Augmentation for Commonsense Reasoning

Yang, Yiben; Malaviya, Chaitanya; Fernandez, Jared; Swayamdipta, Swabha; Bras, Ronan Le; Wang, Jiping; Bhagavatula, Chandra; Choi, Yejin; Downey, Doug

doi:10.18653/v1/2020.findings-emnlp.90

Cited by 83 publications

(75 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work focuses on extractive question-answering (QA), which motivates the need for different generative models. Yang et al (2020) filter generated examples using influence functions, or methods that attempt to maximise diversity; we find that a different approach that considers answer agreement between QA models trained with different random seeds leads to better performance in our setting.…”

Section: Synthetic Question Generationmentioning

confidence: 96%

See 1 more Smart Citation

Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation

Bartolo¹,

Thrush²,

Jia³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Despite recent progress, state-of-the-art question answering models remain vulnerable to a variety of adversarial attacks. While dynamic adversarial data collection, in which a human annotator tries to write examples that fool a model-in-the-loop, can improve model robustness, this process is expensive which limits the scale of the collected data. In this work, we are the first to use synthetic adversarial data generation to make question answering models more robust to human adversaries. We develop a data generation pipeline that selects source passages, identifies candidate answers, generates questions, then finally filters or relabels them to improve quality. Using this approach, we amplify a smaller human-written adversarial dataset to a much larger set of synthetic question-answer pairs. By incorporating our synthetic data, we improve the state-of-theart on the AdversarialQA dataset by 3.7F 1 and improve model generalisation on nine of the twelve MRQA datasets. We further conduct a novel human-in-the-loop evaluation and show that our models are considerably more robust to new human-written adversarial examples: crowdworkers can fool our model only 8.8% of the time on average, compared to 17.6% for a model trained without synthetic data.

show abstract

Section: Synthetic Question Generationmentioning

confidence: 96%

“…We use influence functions (Cook and Weisberg, 1982;Koh and Liang, 2017) to estimate the effect on the validation loss of including a synthetic example as explored by Yang et al (2020), but adapted for QA. We filter out examples estimated to increase the validation loss.…”

Section: Influence Functionsmentioning

confidence: 99%

Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation

Bartolo¹,

Thrush²,

Jia³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…While data augmentation (DA) has been widely adopted in computer vision (Shorten & Khoshgoftaar, 2019), DA for language tasks is less straightforward. Recently, generative language models have been used to synthesize examples for various NLP tasks (Kumar et al, 2020;Anaby-Tavor et al, 2020;Puri et al, 2020;Yang et al, 2020). Different from these methods which focus on the low-resource language-only tasks, our method demonstrates the advantage of synthetic captions in large-scale vision-language pre-training.…”

Section: Data Augmentationmentioning

confidence: 99%

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Li¹,

Li²,

Xiong³

et al. 2022

Preprint

139

View full text Add to dashboard Cite

Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.

show abstract

“…More recent work has used word embeddings (Wang and Yang, 2015) and LSTM language models (Fadaee et al, 2017) to perform word replacement. Other methods focus on fine-tuning contextual language models (Kobayashi, 2018;Wu et al, 2019b;Kumar et al, 2020) or large generative models (Anaby-Tavor et al, 2020;Yang et al, 2020;Kumar et al, 2020) to generate synthetic examples.…”

Section: Data Augmentation In Nlpmentioning

confidence: 99%

SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

Ng¹,

Cho

Ghassemi

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Models that perform well on a training domain often fail to generalize to out-of-domain (OOD) examples. Data augmentation is a common method used to prevent overfitting and improve OOD generalization. However, in natural language, it is difficult to generate new examples that stay on the underlying data manifold. We introduce SSMBA, a data augmentation method for generating synthetic training examples by using a pair of corruption and reconstruction functions to move randomly on a data manifold. We investigate the use of SSMBA in the natural language domain, leveraging the manifold assumption to reconstruct corrupted text with masked language models. In experiments on robustness benchmarks across 3 tasks and 9 datasets, SSMBA consistently outperforms existing data augmentation methods and baseline models on both in-domain and OOD data, achieving gains of 0.8% accuracy on OOD Amazon reviews, 1.8% accuracy on OOD MNLI, and 1.4 BLEU on in-domain IWSLT14 German-English. 1

show abstract

Generative Data Augmentation for Commonsense Reasoning

Cited by 83 publications

References 50 publications

Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation

Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

Contact Info

Product

Resources

About