Query Embedding Pruning for Dense Retrieval

Tonellotto, Nicola; Macdonald, Craig

doi:10.1145/3459637.3482162

Cited by 17 publications

(6 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [133], the authors propose a method called DensePhrases, which uses phrase retrieval to learn passage retrieval. In [134], the authors propose a method for pruning query embeddings to improve dense retrieval performance. In [135], the authors propose a method for learning multi-view document representations to improve open-domain dense retrieval.…”

Section: Multi-vector Representationmentioning

confidence: 99%

Information Retrieval: Recent Advances and Beyond

Hambarde¹,

Proença²

2023

Preprint

View full text Add to dashboard Cite

In this paper, we provide a detailed overview of the models used for information retrieval in the first and second stages of the typical processing chain. We discuss the current state-of-the-art models, including methods based on terms, semantic retrieval, and neural. Additionally, we delve into the key topics related to the learning process of these models. This way, this survey offers a comprehensive understanding of the field and is of interest for for researchers and practitioners entering/working in the information retrieval domain.

show abstract

Section: Multi-vector Representationmentioning

confidence: 99%

Information Retrieval: Recent Advances and Beyond

Hambarde¹,

Proença²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…in the IR setting. Tonellotto and Macdonald [48] prune the embeddings of the query terms, however they do not reduce the number of embeddings for the documents in the index.…”

Section: Related Workmentioning

confidence: 99%

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

Hofstätter¹,

Khattab²,

Althammer³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent progress in neural information retrieval has demonstrated large gains in effectiveness, while often sacrificing the efficiency and interpretability of the neural model compared to classical approaches. This paper proposes ColBERTer, a neural retrieval model using contextualized late interaction (ColBERT) with enhanced reduction. Along the effectiveness Pareto frontier, ColBERTer's reductions dramatically lower ColBERT's storage requirements while simultaneously improving the interpretability of its token-matching scores. To this end, ColBERTer fuses single-vector retrieval, multivector refinement, and optional lexical matching components into one model. For its multi-vector component, ColBERTer reduces the number of stored vectors per document by learning unique whole-word representations for the terms in each document and learning to identify and remove word representations that are not essential to effective scoring. We employ an explicit multi-task, multi-stage training to facilitate using very small vector dimensions. Results on the MS MARCO and TREC-DL collection show that ColBERTer can reduce the storage footprint by up to 2.5×, while maintaining effectiveness. With just one dimension per token in its smallest setting, ColBERTer achieves index storage parity with the plaintext size, with very strong effectiveness results. Finally, we demonstrate ColBERTer's robustness on seven high-quality outof-domain collections, yielding statistically significant gains over traditional retrieval baselines.does doxycycline contain sulfa BERT tokenized (9 subword-tokens): 'does', 'do', '##xy', '##cy', '##cl', '##ine', 'contain', 'sul', '##fa'ColBERTer BOW 2 (30 saved vectors from 84 subword-tokens): photosensitivity doxycycline 12.9 sulfa 14.2 sunburned rash clothing sunlight allergic compound drugs containing 6.6 take safely wear . is no 4.7 exposed ... Fulltext: No doxycycline is not a sulfa containing compound, so you may take it safely if you are allergic to sulfa drugs. You should be aware, however, that doxycycline may cause photosensitivity, so you should wear appropriate clothing, or you may get easily sunburned or develop a rash if you are exposed to sunlight.

show abstract

“…embeddings in the top-ranked documents can be added to the query as a form of pseudo-relevance feedback (PRF) [28]). Furthermore, the per-token embeddings are interpretable [20], and exhibit correlations with IDF [5,27]. On the other hand, access to the voluminous accurate document embeddings in main memory required for the efficient application of the second stage is a major drawback of ColBERT.…”

Section: Multiple Representation Dense Retrievalmentioning

confidence: 99%