Finding Generalizable Evidence by Learning to Convince Q&A Models

Abstract

We propose a system that finds the strongest supporting evidence for a given answer to a question, using passage-based questionanswering (QA) as a testbed. We train evidence agents to select the passage sentences that most convince a pretrained QA model of a given answer, if the QA model received those sentences instead of the full passage. Rather than finding evidence that convinces one model alone, we find that agents select evidence that generalizes; agent-chosen evidence increases the plausibility of the supported answer, as judged by other QA models and humans. Given its general nature, this approach improves QA in a robust manner: using agentselected evidence (i) humans can correctly answer questions with only ∼20% of the full passage and (ii) QA models can generalize to longer passages and harder questions. . 2018a. Evidence extraction for machine reading comprehension with deep probabilistic logic. CoRR, abs/1902.08852.

Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering

“…The BERT-QA baseline scores surprisingly low. A possible explanation is that, in the original setting, Perez et al (2019)'s model learned to spot a (usually) single relevant sentence among a passage of irrelevant sentences. In our setting, though, all the chains are partially relevant, making it harder for the model to distinguish just one as central.…”

“…In the context of QA, there are multiple notions of explanation/justification, including showing an authoritative, answer-bearing sentence (Perez et al, 2019), a collection of text snippets supporting an answer (DeYoung et al, 2020), an attention map over a passage (Seo et al, 2016), a synthesized phrase connecting question and answer (Rajani et al, 2019), or the syntactic pattern used to locate the answer (Ye et al, 2020;Hancock et al, 2018). These methods are primarily designed for answers to "lookup" questions, to explain where and how an answer was found in a corpus.…”

“…We also consider a baseline, BERT-QA, by adapting the approach of Perez et al (2019) to our task. In the original work, given a passage of text and a multiple choice question, the system identifies the sentence(s) S that are the most convincing evidence for a given answer option a i .…”

Frustratingly Hard Evidence Retrieval for QA Over Books

“…One way we believe that can improve the distant supervision signals is by iteratively updating the ranker and reader like in Hard-EM (Min et al, 2019;Guu et al, 2020). Another possible direction is to extend the idea of inferring evidence on training data with game-theoretic approaches (Perez et al, 2019;Feng et al, 2020), then use the inferred evidence paragraph as labels to train the ranker.…”

Learning to Recover Reasoning Chains for Multi-Hop Question Answering via Cooperative Games

“…Our work is the first to recover reasoning chains in a more general unsupervised setting, thus falling into the direction of denoising over distant supervised signals. From this perspective, the most relevant studies in the NLP field includes Wang, Yu, Guo, Wang, Klinger, Zhang, Chang, Tesauro, Zhou, and Jiang [21] and Min, Chen, Hajishirzi, and Zettlemoyer [22] for evidence identification in opendomain QA and Lei, Barzilay, and Jaakkola [5] and Perez, Karamcheti, Fergus, Weston, Kiela, and Cho [23] for rationale recovery.…”