Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation

Abstract

A major obstacle to the wide-spread adoption of neural retrieval models is that they require large supervised training sets to surpass traditional term-based techniques, which are constructed from raw corpora. In this paper, we propose an approach to zero-shot learning for passage retrieval that uses synthetic question generation to close this gap. The question generation system is trained on general domain data, but is applied to documents in the targeted domain. This allows us to create arbitrarily large, yet noisy, question-passage relevance pairs that are domain specific. Furthermore, when this is coupled with a simple hybrid termneural model, first-stage retrieval performance can be improved further. Empirically, we show that this is an effective strategy for building neural passage retrieval models in the absence of large training corpora. Depending on the domain, this technique can even approach the accuracy of supervised models.

Neural Passage Retrieval with Improved Negative Contrast

“…In this work, we adopt the approach mentioned in (Ma et al, 2020), and trained our own question generation model based on Text-To-Text Transfer Transformer (T5) (Raffel et al, 2020). We then use this model to generate synthetic questions for the passages in a collection of documents.…”

“…Training an effective neural retrieval model usually requires a large amount of high-quality data. To alleviate the need of high-quality data, training can be approached in two-stages: pre-training on noise data (Guo et al, 2018;Chang et al, 2020;Ma et al, 2020) and fine tuning on a smaller amount of highquality data, also regarded as "gold" data.…”

“…We first use hard negatives on the data that we use to pre-train the models. We leverage the question generator model described in (Ma et al, 2020) and generate new questions for each of the passages we use in the pre-training stage (Stage 1). During pre-training we use negatives generated from strategy 4 2 to improve the retrieval model, as the other strategies could introduce more false negatives into the data.…”

GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

“…GPL improves the performance by up to 8.9 points nDCG@10 compared to state-of-the-art model trained solely on MS MARCO. Compared to the previous state-of-the-art domain-adaption method QGen (Ma et al, 2021;Thakur et al, 2021b), GPL improves the performance by up to 5.2 nDCG@10 points. Training with GPL is easy, fast, and data efficient.…”

“…This performed well in the zero-shot retrieval benchmark BeIR (Thakur et al, 2021b). Ma et al (2021) propose QGen, that uses a query generator trained on general domain data to synthesize domain-targeted queries for the target corpus, on which a dense retriever is trained from scratch. Following this idea, Thakur et al (2021b) views QGen as a post-training method to adapt powerful MS MARCO retrievers to the target domains.…”