Is Retriever Merely an Approximator of Reader?

Yang, Sohee; Seo, Minjoon

doi:10.48550/arxiv.2010.10999

Cited by 17 publications

(21 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The major improvements in this work are attributed to end-to-end training, which amounts to a type of distillation from the powerful T5 model into the retrieval model. It's interesting to compare this to more direct distillation methods (Izacard and Grave, 2020;Yang and Seo, 2020), which also re-ported similar gains. Our method also relies on a reader model indirectly, through the global filtering stage of generated questions in PAQ.…”

Section: Pretraining For Retrievalmentioning

confidence: 90%

Domain-matched Pre-training Tasks for Dense Retrieval

Oğuz¹,

Lakhotia²,

Gupta³

et al. 2021

Preprint

View full text Add to dashboard Cite

Pre-training on larger datasets with ever increasing model size is now a proven recipe for increased performance across almost all NLP tasks. A notable exception is information retrieval, where additional pre-training has so far failed to produce convincing results. We show that, with the right pre-training setup, this barrier can be overcome. We demonstrate this by pre-training large bi-encoder models on 1) a recently released set of 65 million synthetically generated questions, and 2) 200 million post-comment pairs from a preexisting dataset of Reddit conversations made available by pushshift.io. We evaluate on a set of information retrieval and dialogue retrieval benchmarks, showing substantial improvements over supervised baselines.

show abstract

Section: Pretraining For Retrievalmentioning

confidence: 90%

Domain-matched Pre-training Tasks for Dense Retrieval

Oğuz¹,

Lakhotia²,

Gupta³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The index consists of 1.2M 128-dimensional INT8 vectors (scalar-quantized and unsigned), which are the dense embeddings of the subset of the 21M passages in Wikipedia, filtered by a RoBERTa [30]-based binary classifier trained with logistic regression to exclude uninformative passages. Positives to train this classifier are the top 200 passages for each question on Natural Questions dataset and EfficientQA development set, retrieved by [40], a DPR retriever further finetuned on hard negatives using knowledge distillation from a DPR reader. Negatives are uniformly drawn from the set of 21M passages, excluding positives.…”

Section: Top Systemsmentioning

confidence: 99%

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Min,

Boyd-Graber,

Alberti

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We review the EfficientQA competition 1 from NeurIPS 2020 2 . The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing large, redundant, retrieval corpora or the parameters of large learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.

show abstract

“…To make the retrieval tractable, the term-level function is approximated to first retrieve an initial set of candidates, which are then re-ranked with the true score. In the context of question answering, knowledge distillation has been used to train retrievers, either using the attention scores of the reader of the downstream task as synthetic labels (Izacard & Grave, 2021), or the relevance score from a cross encoder (Yang & Seo, 2020). Luan et al (2020) compares, theoretically and empirically, the performance of sparse and dense retrievers, including bi-, cross-and poly-encoders.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Dense Information Retrieval with Contrastive Learning

Izacard¹,

Caron²,

Hosseini³

et al. 2021

Preprint

View full text Add to dashboard Cite

Information retrieval is an important component in natural language processing, for knowledge intensive tasks such as question answering and fact checking. Recently, information retrieval has seen the emergence of dense retrievers, based on neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new domains or applications with no training data, and are often outperformed by term-frequency methods such as BM25 which are not supervised. Thus, a natural question is whether it is possible to train dense retrievers without supervision. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers, and show that it leads to strong retrieval performance. More precisely, we show on the BEIR benchmark that our model outperforms BM25 on 11 out of 15 datasets. Furthermore, when a few thousands examples are available, we show that fine-tuning our model on these leads to strong improvements compared to BM25. Finally, when used as pre-training before fine-tuning on the MS-MARCO dataset, our technique obtains state-of-the-art results on the BEIR benchmark.

show abstract

Is Retriever Merely an Approximator of Reader?

Cited by 17 publications

References 14 publications

Domain-matched Pre-training Tasks for Dense Retrieval

Domain-matched Pre-training Tasks for Dense Retrieval

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Unsupervised Dense Information Retrieval with Contrastive Learning

Contact Info

Product

Resources

About