Do Answers to Boolean Questions Need Explanations? Yes

Abstract

Existing datasets that contain boolean questions, such as BoolQ and TYDI QA , provide the user with a YES/NO response to the question. However, a one word response is not sufficient for an explainable system. We promote explainability by releasing a new set of annotations marking the evidence in existing TYDI QA and BoolQ datasets. We show that our annotations can be used to train a model that extracts improved evidence spans compared to models that rely on existing resources. We confirm our findings with a user study which shows that our extracted evidence spans enhance the user experience. We also provide further insight into the challenges of answering boolean questions, such as passages containing conflicting YES and NO answers, and varying degrees of relevance of the predicted evidence.

GAAMA 2.0: An Integrated System that Answers Boolean and Extractive Question

“…Other QA demos include: ARES (Ferritto et al, 2020a), a QA demo that features ensembling of several MRC systems, BERTSerini (Yang et al, 2019) , which leverages the Anserini IR toolkit (Yang et al, 2017) to extract relevant documents given a question in English only, NAMER (Zhang et al, 2021b) for multi-hop knowledge base QA, and Talk to Papers (Zhao and Lee, 2020) for QA in academic search. We make use of two datasets in developing our system: TYDI QA (Clark et al, 2020) and BoolQ-X (Rosenthal et al, 2021). TYDI QA is a multilingual MRC dataset containing questions in multiple languages.…”

“…This also mimics what occurs during real-world use of this component. In addition, we supplemented the TYDI QA training data with questions and 200 word passages selected from BoolQ-X (Rosenthal et al, 2021), which is more compatible with the MRC task and the TYDI QA data than BoolQ.…”