♫ MuSiQue: Multihop Questions via Single-hop Question Composition

Abstract

Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts. Can we create a question answering (QA) dataset that, by construction, requires proper multihop reasoning? To this end, we introduce a bottom–up approach that systematically selects composable pairs of single-hop questions that are connected, that is, where one reasoning step critically relies on information from another. This bottom–up methodology lets us explore a vast space of questions and add stringent filters as well as other mechanisms targeting connected reasoning. It provides fine-grained control over the construction process and the properties of the resulting k-hop questions. We use this methodology to create MuSiQue-Ans, a new multihop QA dataset with 25K 2–4 hop questions. Relative to existing datasets, MuSiQue-Ans is more difficult overall (3× increase in human–machine gap), and harder to cheat via disconnected reasoning (e.g., a single-hop model has a 30-point drop in F1). We further add unanswerable contrast questions to produce a more stringent dataset, MuSiQue-Full. We hope our datasets will help the NLP community develop models that perform genuine multihop reasoning.1

Hey AI, Can You Solve Complex Tasks by Talking to Agents?

“…COMMAQA-E: Explicit Decomposition. This dataset consists of multi-hop questions from the movie domain where the reasoning needed to answer the question is Explicitly described in the question itself (Yang et al, 2018;Ho et al, 2020;Trivedi et al, 2021). For example, "What awards have the movies directed by Spielberg won?".…”

Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering

“…HotpotQA (Yang et al, 2018) and MuSiQue (Trivedi et al, 2022) add complexity to reading comprehension by introducing multi-hop questions where the answer requires reasoning over two documents, but neither of these datasets naturally elicit their questions. HotpotQA pre-selects two Wikipedia passages and asks workers to write questions using both passages, and MuSiQue composes multi-hop questions from existing single hop questions.…”

Teaching Broad Reasoning Skills via Decomposition-Guided Contexts

“…To enable development of better systems, several recent multihop QA datasets come with question decompostion annotations Talmor and Berant, 2018;Geva et al, 2021;Trivedi et al, 2022;. These works have enabled the development of explicit multistep reasoning systems that first decomposes a multihop question into sub-questions, and answers the subquestions step-by-step to arrive at the answer (Min et al, 2019b;Trivedi et al, 2022). Our goal in this work is to use decompositions to instead teach black-box language to perform multistep reasoning implicitly (within the model).…”

“…Jiang and Bansal (2019); Ding et al (2021) created adversarial multihop question by perturbing the reasoning chains in HotpotQA . Other datasets (Trivedi et al, 2020(Trivedi et al, , 2022Lee et al, 2021) ensure robust reasoning via minimally perturbed unanswerable questions. Our approach targets a broader set of questions and eliminates multiple reasoning shortcuts.…”

“…A second strand has shown that perturbing the existing (natural) contexts of RC instances in a targeted fashion can reduce artifact-based reasoning (Jia and Liang, 2017;Trivedi et al, 2020). A third strand has shown that carefully constructing contexts (for synthetic questions) to have sufficient distractors can reduce artifacts exploitable by current models (Trivedi et al, 2022;.…”