NA‐Reviewer: Reviewing the context to improve the error accumulation issue for multi‐hop QA

Fu, Ruiliu; Wang, Han; Zhou, Jun; Zhang, Xuejun

doi:10.1049/ell2.12411

Cited by 2 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CRERC (Fu et al, 2021a) is a pipeline model that includes three modules: relation extractor, reader, and comparator. NA-Reviewer (Fu et al, 2021b) is an improved version of CRERC, as it addresses the error accumulation issue. It is noted that both CRERC and NA-Reviewer models are evaluated on only 2Wiki.…”

Section: B3 Results Comparisonmentioning

confidence: 99%

Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering

Ho,

Duong Nguyen,

Sugawara

et al. 2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

To explain the predicted answers and evaluate the reasoning abilities of models, several studies have utilized underlying reasoning (UR) tasks in multi-hop question answering (QA) datasets. However, it remains an open question as to how effective UR tasks are for the QA task when training models on both tasks in an endto-end manner. In this study, we address this question by analyzing the effectiveness of UR tasks (including both sentence-level and entitylevel tasks) in three aspects: (1) QA performance, (2) reasoning shortcuts, and (3) robustness. While the previous models have not been explicitly trained on an entity-level reasoning prediction task, we build a multi-task model that performs three tasks together: sentencelevel supporting facts prediction, entity-level reasoning prediction, and answer prediction. Experimental results on 2WikiMultiHopQA and HotpotQA-small datasets reveal that (1) UR tasks can improve QA performance. Using four debiased datasets that are newly created, we demonstrate that (2) UR tasks are helpful in preventing reasoning shortcuts in the multi-hop QA task. However, we find that (3) UR tasks do not contribute to improving the robustness of the model on adversarial questions, such as sub-questions and inverted questions. We encourage future studies to investigate the effectiveness of entity-level reasoning in the form of natural language questions (e.g., sub-question forms). 1 * Equal contribution. 1 Our data and code are available at https://github. com/Alab-NII/multi-hop-analysis Question: Who is the paternal grandfather of Joan of Valois, Countess of Beaumont? Paragraph A: Joan of Valois, Countess of Beaumont [1] Joan of Valois (1304 -1363) was the daughter of Charles of Valois and his second wife ... Paragraph B: Charles, Count of Valois [2] Charles of Valois (12 March 1270 -16 December 1325), the third son of Philip III of France and, ... . [3] ... Answer: Philip III of France a) Standard QA task format b) UR tasks and three aspects Sentence-level supporting facts: 1, 2Entity-level reasoning prediction (Evidence):Step 1: ("Joan of Valois, Countess of Beaumont", "father", "Charles of Valois") &Step 2: ("Charles of Valois", "father", "Philip III of France") QA Performance Reasoning Shortcuts Robustness Paragraph A: Joan of Valois, Countess of Beaumont [1] We can also establish the global weak solution ... [2] Joan of Valois (1304 -1363) was the daughter of Charles of Valois and his second wife ... Paragraph B: Charles, Count of Valois [3] This gives a clear impulse to develop ... [4] Charles of Valois (12 March 1270 -16 December 1325), the third son of Philip III of France and, ... . [5] ...

show abstract

Section: B3 Results Comparisonmentioning

confidence: 99%

Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering

Ho,

Duong Nguyen,

Sugawara

et al. 2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

show abstract

“…EX(SA) (Trivedi et al, 2022b) decomposes a complex question into single-hop questions and builds a directed acyclic graph (DAG) for each singlehop reader (SA) to memorize the answer flow. NA-Reviewer (Fu et al, 2022) proposes a reviewer model that can fix the error prediction from incorrect evidence. We include FiD (Izacard and 2021) as the baseline End2End reader model.…”

Section: Baselinesmentioning

confidence: 99%

HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution

Liu,

Yavuz,

Meng

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

The dominant paradigm of textual question answering with end-to-end neural models excels at answering simple questions but falls short on explainability and dealing with more complex questions. This stands in contrast to the broad adaptation of semantic parsing approaches over structured data sources (e.g., relational database), that convert questions to logical forms and execute them with query engines. Towards the goal of combining the strengths of neural and symbolic methods, we propose a framework of question parsing and execution for textual QA. It comprises two central pillars: (1) parsing a question of varying complexity into an intermediate representation, named H-expression, which symbolically represents how an answer to the question can be reached by hierarchically combining answers from the primitive simple questions; (2) to execute the resulting expression, we design a hybrid executor, which integrates deterministic rules to translate the symbolic operations with a drop-in neural reader to answer each simple question. The proposed framework can be viewed as a top-down question parsing followed by a bottom-up answer backtracking. H-expressions closely guide the execution process, offering higher precision besides better interpretability while still preserving the advantages of the neural readers for resolving primitive elements. Our extensive experiments on four different QA datasets show that the proposed framework outperforms existing approaches in supervised, few-shot, and zero-shot settings, while also effectively exposing the underlying reasoning process 1 .

show abstract

NA‐Reviewer: Reviewing the context to improve the error accumulation issue for multi‐hop QA

Cited by 2 publications

References 12 publications

Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering

Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering

HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution

Contact Info

Product

Resources

About