Synthetic Target Domain Supervision for Open Retrieval QA

Abstract

Neural passage retrieval is a new and promising approach in open retrieval question answering. In this work, we stress-test the Dense Passage Retriever (DPR)-a state-of-the-art (SOTA) open domain neural retrieval model-on closed and specialized target domains such as COVID-19, and find that it lags behind standard BM25 in this important real-world setting. To make DPR more robust under domain shift, we explore its fine-tuning with synthetic training examples, which we generate from unlabeled target domain text using a text-to-text generator. In our experiments, this noisy but fully automated target domain supervision gives DPR a sizable advantage over BM25 in out-of-domain settings, making it a more viable model in practice. Finally, an ensemble of BM25 and our improved DPR model yields the best results, further pushing the SOTA for open retrieval QA on multiple out-of-domain test sets.

Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrieval

“…We call this the unconditioned question generator, since the questions are not conditioned to be about any specific entities. This serves as a baseline question generation approach and is comparable with prior work [5,14,17] in synthetic data generation for IR, which do not enforce such specific conditioning into the question generation process.…”

“…To compare our approach with a generation strategy that does not use any conditioning, we also train an unconditioned generation system, similar to Reddy et al [17], that generates question-answer pairs using just the passage as input. We call this the unconditioned question generator, since the questions are not conditioned to be about any specific entities.…”

“…Our approach follows this line of work by leveraging the attentions of an IR model over given passages as a signal for better synthetic data augmentation. Prior work has also explored synthetic question generation for both question answering [1,18,19] and neural information retrieval [5,14,17]; different approaches to generating questions from passages include: (a) unconditioned generation [5,17], (b) generation conditioned on the candidate answer phrases within the passage [1,18], and (c) conditioned on the summary of the passage [13,24]. In contrast, our approach generates questions that are targeted towards the deficiencies of a given neural IR model, by conditioning the generation on sparsely attended entities in the passage.…”

InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers

“…In addition, to prompt-based generation of training data, there are multiple proposals for self-supervised adaptation of out-of-domain models using generative pseudo-labeling [22,38,51]. To this end, questions or queries are generated using a pretrained seq2seq model (though an LLMs can be used as well) and negative examples are mined using either BM25 or an out-of-domain retriever or ranker.…”