End-to-End Training of Neural Retrievers for Open-Domain Question Answering

Abstract

Recent work on training neural retrievers foropen-domain question answering (OpenQA) has employed both supervised and unsupervised approaches. However, it remains unclear how unsupervised and supervised methods can be used most effectively for neural retrievers. In this work, we systematically study retriever pre-training. We first propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans, followed by supervised finetuning using question-context pairs. This approach leads to absolute gains of 2+ points over the previous best result in the top-20 retrieval accuracy on Natural Questions and TriviaQA datasets. We next explore two approaches for end-toend training of the reader and retriever components in OpenQA models, which differ in the manner the reader ingests the retrieved documents. Our experiments demonstrate the effectiveness of these approaches as we obtain state-of-the-art results. On the Natural Questions dataset, we obtain a top-20 retrieval accuracy of 84%, an improvement of 5 points over the recent DPR model. We also showcase good results on answer extraction, outperforming recent models such as REALM and RAG by 3+ points. Our code is available at: https: //github.com/NVIDIA/Megatron-LM. * This work was done during an internship at NVIDIA.

Hindsight: Posterior-guided training of retrievers for improved open-ended generation

“…While this is a valid mechanism, perhaps conditioning on individual passages like we do is more precise for relevance supervision. Indeed, recent work (Sachan et al, 2021) illustrates by using Fusion-in-Decoder during inference but foregoing the decoder's attention weights and using an equivalent version of the MARGINALIZEDLOSS for training the retriever. Furthermore, Fusion-in-Decoder is uniquely useful for QA style tasks, where it has to select the correct answer from many passages.…”

Learning to Retrieve Passages without Supervision

“…Pretraining for dense retrieval has recently gained a considerable attention, following the success of self-supervised models in many NLP tasks Liu et al, 2019;Brown et al, 2020). While most works focus on fine-tuning such retrievers on large datasets after pretraining Guu et al, 2020;Sachan et al, 2021;Gao and Callan, 2021a), we attempt to bridge the gap between unsupervised dense models and strong sparse (e.g., BM25; Robertson and Zaragoza 2009) or supervised dense baselines (e.g., DPR; Karpukhin et al 2020). A concurrent work by Oguz et al (2021) presented DPR-PAQ, which shows strong results on NQ after pretraining.…”

“…This is done mainly to avoid uninformative recurring words, e.g., verbs or adjectives. Note that as opposed to other approaches for span filtering (Glass et al, 2020;Guu et al, 2020;Sachan et al, 2021), our heuristics do not require any model.…”

Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answering

“…Most of the OpenQA models also consist of a retriever and a reasoner. The retriever is devised as a sparse term-based method such as BM25 (Robertson and Zaragoza, 2009) or a trainable dense passage retrieval method (Karpukhin et al, 2020;Sachan et al, 2021a), and the reasoner deals with each doc-ument individually (Guu et al, 2020) or fuses all the documents together (Izacard and Grave, 2021). Different from the documents in openQA, the subgraphs in KBQA can be only obtained by multi-hop retrieval and the reasoner should deal with the entire subgraph instead of each individual relation to find the answer.…”