Learning to Ask Unanswerable Questions for Machine Reading Comprehension

Zhu, Haichao; Dong, Li; Wei, Furu; Wang, Wenhui; Qin, Bing; Liu, Ting

doi:10.18653/v1/p19-1415

Cited by 35 publications

(32 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hu et al (2019) address unanswerability of questions from a given text using additional verification steps. Other approaches have shown the benefit of synthetic data to improve performance in SQuAD2.0 (Zhu et al, 2019;Alberti et al, 2019). In contrast to prior work, we demonstrate that despite improving performance on test sets that include unanswerable questions, the problem persists when adversarially choosing from a larger space of questions.…”

Section: Concurrent Work On Modelcontrasting

confidence: 71%

Undersensitivity in Neural Reading Comprehension

Welbl¹,

Minervini²,

Bartolo³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Current reading comprehension methods generalise well to in-distribution test sets, yet perform poorly on adversarially selected data. Prior work on adversarial inputs typically studies model oversensitivity: semantically invariant text perturbations that cause a model's prediction to change. Here we focus on the complementary problem: excessive prediction undersensitivity, where input text is meaningfully changed but the model's prediction does not, even though it should. We formulate an adversarial attack which searches among semantic variations of the question for which a model erroneously predicts the same answer, and with even higher probability. We demonstrate that models trained on both SQuAD2.0 and NewsQA are vulnerable to this attack, and then investigate data augmentation and adversarial training as defences. Both substantially decrease adversarial vulnerability, which generalises to held-out data and held-out attack spaces. Addressing undersensitivity furthermore improves model robustness on the previously introduced ADDSENT and ADDONE-SENT datasets, and models generalise better when facing train/evaluation distribution mismatch: they are less prone to overly rely on shallow predictive cues present only in the training set, and outperform a conventional model by as much as 10.9% F 1 .

show abstract

Section: Concurrent Work On Modelcontrasting

confidence: 71%

Undersensitivity in Neural Reading Comprehension

Welbl¹,

Minervini²,

Bartolo³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

show abstract

“…As our goal is to provide a broad suite of questions that test a single model's reading ability, we additionally provide synthetic augmentations to some of the datasets in our evaluation server. Several recent papers have proposed question transformations that result in out-of-distribution test examples, helping to judge the generalization capability of reading models (Ribeiro et al, 2018(Ribeiro et al, , 2019Zhu et al, 2019). We collect the best of these, add some of our own, and keep those that generate reasonable and challenging questions.…”

Section: Introductionmentioning

confidence: 99%

Comprehensive Multi-Dataset Evaluation of Reading Comprehension

Dua¹,

Gottumukkala²,

Talmor³

et al. 2019

Proceedings of the 2nd Workshop on Machine Reading for Question Answering

View full text Add to dashboard Cite

Reading comprehension is one of the crucial tasks for furthering research in natural language understanding. A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context. Given the availability of many such datasets, comprehensive and reliable evaluation is tedious and time-consuming for researchers working on this problem. We present an evaluation server, ORB, that reports performance on seven diverse reading comprehension datasets, encouraging and facilitating testing a single model's capability in understanding a wide variety of reading phenomena. The evaluation server places no restrictions on how models are trained, so it is a suitable test bed for exploring training paradigms and representation learning for general reading facility. As more suitable datasets are released, they will be added to the evaluation server. We also collect and include synthetic augmentations for these datasets, testing how well models can handle out-of-domain questions.

show abstract

“…When we extensively tested the Question Answering System keeping in mind how the answer is generated, it is found that Question Comprehension plays a significant role in the question answering system [38]. Also, systems like [46] introduce a pair-to-sequence model that captures the interaction between the question asked and the given paragraph. Specific systems like ParaQG [20] try to generate the questions from the paragraph.…”

Section: Related Workmentioning

confidence: 99%

Automatic question-answer pairs generation and question similarity mechanism in question answering system

2021

View full text Add to dashboard Cite

With the swift growth of the information over the past few years, taking full benefit is increasingly essential. Question Answering System is one of the promising methods to access this much information. The Question Answering System lacks humans’ common sense and reasoning power and cannot identify unanswerable questions and irrelevant questions. These questions are answered by making unreliable and incorrect guesses. In this paper, we address this limitation by proposing a Question Similarity mechanism. Before a question is posed to a Question-Answering system, it is compared with possible generated questions of the given paragraph, and then a Question Similarity Score is generated. The Question Similarity mechanism effectively identifies the unanswerable and irrelevant questions. The proposed Question Similarity mechanism incorporates a human way of reasoning to identify unanswerable and irrelevant questions. This mechanism can avoid the unanswerable and irrelevant questions altogether from being posed to the Question Answering system. It helps the Question Answering Systems to focus only on the answerable questions to improve their performance. Along with this, we introduce an application of the Question Answering System that generates the question-answer pairs given a passage and is useful in several fields.

show abstract

Learning to Ask Unanswerable Questions for Machine Reading Comprehension

Cited by 35 publications

References 32 publications

Undersensitivity in Neural Reading Comprehension

Undersensitivity in Neural Reading Comprehension

Comprehensive Multi-Dataset Evaluation of Reading Comprehension

Automatic question-answer pairs generation and question similarity mechanism in question answering system

Contact Info

Product

Resources

About