Learning with Instance Bundles for Reading Comprehension

Dua, Dheeru; Dasigi, Pradeep; Singh, Sameer; Gardner, Matt

doi:10.48550/arxiv.2104.08735

Cited by 3 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Last, we showed that constraint sets are useful for evaluation. Future work can use constraints as a supervision signal, similar to Dua et al (2021), who leveraged dependencies between training examples to enhance model performance.…”

Section: Discussionmentioning

confidence: 99%

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition

Geva

Wolfson

Berant

2022

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Recent efforts to create challenge benchmarks that test the abilities of natural language understanding models have largely depended on human annotations. In this work, we introduce the “Break, Perturb, Build” (BPB) framework for automatic reasoning-oriented perturbation of question-answer pairs. BPB represents a question by decomposing it into the reasoning steps that are required to answer it, symbolically perturbs the decomposition, and then generates new question-answer pairs. We demonstrate the effectiveness of BPB by creating evaluation sets for three reading comprehension (RC) benchmarks, generating thousands of high-quality examples without human intervention. We evaluate a range of RC models on our evaluation sets, which reveals large performance gaps on generated examples compared to the original data. Moreover, symbolic perturbations enable fine-grained analysis of the strengths and limitations of models. Last, augmenting the training data with examples generated by BPB helps close the performance gaps, without any drop on the original data distribution.

show abstract

Section: Discussionmentioning

confidence: 99%

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition

Geva

Wolfson

Berant

2022

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Aforementioned work built dataset baselines with popular entity-based PLMs, and thus leave significant performance gaps compared with human evaluation. Asai and Hajishirzi (2020), Dua et al (2021) and Shang et al (2021) leverage features of closely related questions to capture temporal difference to deal with certain types of event-centric questions. Compared to the existing works, we target to various types of event-centric questions.…”

Section: Related Workmentioning

confidence: 99%

Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformation

Lu¹,

Tan²,

Pergola³

et al. 2022

Findings of the Association for Computational Linguistics: EMNLP 2022

View full text Add to dashboard Cite

Human reading comprehension often requires reasoning of event semantic relations in narratives, represented by Event-centric Question-Answering (QA). To address event-centric QA, we propose a novel QA model with contrastive learning and invertible event transformation, call TranCLR. Our proposed model utilizes an invertible transformation matrix to project semantic vectors of events into a common event embedding space, trained with contrastive learning, and thus naturally inject event semantic knowledge into mainstream QA pipelines. The transformation matrix is finetuned with the annotated event relation types between events that occurred in questions and those in answers, using event-aware question vectors. Experimental results on the Event Semantic Relation Reasoning (ESTER) dataset show significant improvements in both generative and extractive settings compared to the existing strong baselines, achieving over 8.4% gain in the token-level F1 score and 3.0% gain in Exact Match (EM) score under the multianswer setting. Qualitative analysis reveals the high quality of the generated answers by TranCLR, demonstrating the feasibility of injecting event knowledge into QA model learning. Our code and models can be found at https://github.com/LuJunru/TranCLR.

show abstract

“…This has led to an increased interest in creating automatic counterfactual data for evaluating outof-distribution generalization (Bowman and Dahl, 2021) and for counterfactual data augmentation (Geva et al, 2021;Longpre et al, 2021). Some work focuses on using heuristics like first-orderlogic (Asai and Hajishirzi, 2020), swapping superlatives and nouns (Dua et al, 2021), or targeting specific data splits (Finegan-Dollak and Verma, 2020). Webster et al (2020) use templates to create large-scale counterfactual data for pre-training to reduce gender bias.…”

Section: Counterfactual Generationmentioning

confidence: 99%

Retrieval-guided Counterfactual Generation for QA

Paranjape¹,

Lamm²,

Tenney³

2021

Preprint

View full text Add to dashboard Cite

Deep NLP models have been shown to learn spurious correlations, leaving them brittle to input perturbations. Recent work has shown that counterfactual or contrastive data -i.e. minimally perturbed inputs -can reveal these weaknesses, and that data augmentation using counterfactuals can help ameliorate them. Proposed techniques for generating counterfactuals rely on human annotations, perturbations based on simple heuristics, and meaning representation frameworks. We focus on the task of creating counterfactuals for question answering, which presents unique challenges related to world knowledge, semantic diversity, and answerability. To address these challenges, we develop a Retrieve-Generate-Filter (RGF) technique to create counterfactual evaluation and training data with minimal human supervision. Using an open-domain QA framework and question generation model trained on original task data, we create counterfactuals that are fluent, semantically diverse, and automatically labeled. Data augmentation with RGF counterfactuals improves performance on out-of-domain and challenging evaluation sets over and above existing methods, in both the reading comprehension and open-domain QA settings. Moreover, we find that RGF data leads to significant improvements in a model's robustness to local perturbations.

show abstract

Learning with Instance Bundles for Reading Comprehension

Cited by 3 publications

References 25 publications

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition

Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformation

Retrieval-guided Counterfactual Generation for QA

Contact Info

Product

Resources

About