Learning to Retrieve Passages without Supervision

Ram, Ori; Shachaf, Gal; Levy, Omer; Berant, Jonathan; Globerson, Amir

doi:10.48550/arxiv.2112.07708

Cited by 4 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[166] proposes GPL method that uses generative pseudo-labeling for unsupervised domain adaptation of dense retrieval. [167] proposes a method called Spider, which enables unsupervised passage retrieval. [168] proposes a method that utilizes contrastive pre-training to learn embeddings for text and code.…”

Section: Multi-vector Representationmentioning

confidence: 99%

Information Retrieval: Recent Advances and Beyond

Hambarde¹,

Proença²

2023

Preprint

View full text Add to dashboard Cite

In this paper, we provide a detailed overview of the models used for information retrieval in the first and second stages of the typical processing chain. We discuss the current state-of-the-art models, including methods based on terms, semantic retrieval, and neural. Additionally, we delve into the key topics related to the learning process of these models. This way, this survey offers a comprehensive understanding of the field and is of interest for for researchers and practitioners entering/working in the information retrieval domain.

show abstract

Section: Multi-vector Representationmentioning

confidence: 99%

Information Retrieval: Recent Advances and Beyond

Hambarde¹,

Proença²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Pre-training. NLP has recently borrowed ideas from contrastive learning techniques in Computer Vision, with the goal of learning high-quality sentence or document representations without annotation [7,11,21,28,29]. The general idea consists in designing pre-training tasks, that are better suited for subsequently training neural retrievers.…”

Section: Distillation Hard Negative Mining and Plm Initializationmentioning

confidence: 99%

From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective

Formal¹,

Lassance²,

Piwowarski³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -a sparse expansion-based retriever -and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained Language Model initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, leading to state-of-the-art results in both scenarios for sufficiently expressive models. CCS CONCEPTS• Information systems → Retrieval models and ranking.

show abstract

“…Retrieval-based language models (R-LMs) have recently been shown to improve over standard neural models in a variety of tasks such as unconditional language modeling (Guu et al, 2018;He et al, 2020), machine translation (Zhang et al, 2018;Gu et al, 2018;Khandelwal et al, 2021), question answering (Karpukhin et al, 2020;Ram et al, 2021), and code generation (Hayati et al, 2018;Hashimoto et al, 2018). The key ingredient of R-LMs is their ability to utilize training examples at test time without having to rely on the information encoded in the model's weights only.…”

Section: Introductionmentioning

confidence: 99%

Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

Alon¹,

Xu²,

He³

et al. 2022

Preprint

View full text Add to dashboard Cite

Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore at test time. While effective, a major bottleneck of using these models in practice is the computationally costly datastore search, which can be performed as frequently as every time step. In this paper, we present RETOMATON -retrieval automatonwhich approximates the datastore search, based on (1) clustering of entries into "states", and (2) state transitions from previous entries. This effectively results in a weighted finite automaton built on top of the datastore, instead of representing the datastore as a flat list. The creation of the automaton is unsupervised, and a RETOMATON can be constructed from any text collection: either the original training corpus or from another domain. Traversing this automaton at inference time, in parallel to the LM inference, reduces its perplexity, or alternatively saves up to 83% of the nearest neighbor searches over kNN-LM (Khandelwal et al., 2020), without hurting perplexity.

show abstract

Learning to Retrieve Passages without Supervision

Cited by 4 publications

References 18 publications

Information Retrieval: Recent Advances and Beyond

Information Retrieval: Recent Advances and Beyond

From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective

Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

Contact Info

Product

Resources

About