SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval

Formal, Thibault; Lassance, Carlos; Piwowarski, Benjamin; Clinchant, Stéphane

doi:10.48550/arxiv.2109.10086

Cited by 40 publications

(84 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also fine-tune the unsupervised model on MS-MARCO and evaluate it on a suite of zero-shot search tasks in the BEIR benchmark (Thakur et al, 2021). In the transfer setting, our models achieve a 5.2% relative improvement over previous methods (Izacard et al, 2021) and is comparable even with methods (Santhanam et al, 2021;Formal et al, 2021;Wang et al, 2020) that demand substantially more computation at test time.…”

Section: Introductionmentioning

confidence: 67%

“…ColBERT v2 (Santhanam et al, 2021) is a multi-vector method that represents the query and the documents as a set of vectors, and employs a multi-step retrieval procedure to obtain relevant documents. Splade v2 (Formal et al, 2021) represents queries and documents as sparse vectors of size equivalent to the vocabulary of the BERT encoder (Devlin et al, 2019). Our cpt-text models compute only one dense embedding per document which are indexed offline and does not depend on any cross-attention re-ranker at query time.…”

Section: Beir Searchmentioning

confidence: 99%

“…cpt-text outperforms previous best embedding methods (Xiong et al, 2020;Hofstätter et al, 2021;Izacard et al, 2021) in both the settings. In the unsupervised setting, BM25 (Robertson, 2009) still achieves the best performance while in the transfer setting cpt-text is competitive with methods that use substantially more compute at test time (Wang et al, 2020;Santhanam et al, 2021;Formal et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Text and Code Embeddings by Contrastive Pre-Training

Neelakantan¹,

Xu²,

Puri³

et al. 2022

Preprint

View full text Add to dashboard Cite

Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.

show abstract

Section: Introductionmentioning

confidence: 67%

Section: Beir Searchmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Text and Code Embeddings by Contrastive Pre-Training

Neelakantan¹,

Xu²,

Puri³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The final representation of the input sequence is then obtained by conducting a pooling operation on the set of |V |-dimensional representations with N elements. For instance, in SPLADEv2 [Formal et al, 2021a], the j-th weight is defined as:…”

Section: Preliminariesmentioning

confidence: 99%

“…Our improved model, which we call SPLADE-mask, is competitive with state-of-the-art lexical retrieval models that incorporate more complex multi-stage training regimes [Mallia et al, 2021, Lin and Ma, 2021, Zhuang and Zuccon, 2021, hard negative mining, or knowledge distillation using pretrained cross-encoders [Formal et al, 2021a].…”

Section: Introductionmentioning

confidence: 99%

Sparsifying Sparse Representations for Passage Retrieval by Top-$k$ Masking

Yang¹,

Ma²,

Lin³

2021

Preprint

View full text Add to dashboard Cite

Sparse lexical representation learning has demonstrated much progress in improving passage retrieval effectiveness in recent models such as DeepImpact, uniCOIL, and SPLADE. This paper describes a straightforward yet effective approach for sparsifying lexical representations for passage retrieval, building on SPLADE by introducing a top-k masking scheme to control sparsity and a self-learning method to coax masked representations to mimic unmasked representations. A basic implementation of our model is competitive with more sophisticated approaches and achieves a good balance between effectiveness and efficiency. The simplicity of our methods opens the door for future explorations in lexical representation learning for passage retrieval.

show abstract

How Different are Pre-trained Transformers for Text Ranking?

Rau

Kamps

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In recent years, large pre-trained transformers have led to substantial gains in performance over traditional retrieval models and feedback approaches. However, these results are primarily based on the MS Marco/TREC Deep Learning Track setup, with its very particular setup, and our understanding of why and how these models work better is fragmented at best. We analyze effective BERT-based cross-encoders versus traditional BM25 ranking for the passage retrieval task where the largest gains have been observed, and investigate two main questions. On the one hand, what is similar? To what extent does the neural ranker already encompass the capacity of traditional rankers? Is the gain in performance due to a better ranking of the same documents (prioritizing precision)? On the other hand, what is different? Can it retrieve effectively documents missed by traditional systems (prioritizing recall)? We discover substantial differences in the notion of relevance identifying strengths and weaknesses of BERT that may inspire research for future improvement. Our results contribute to our understanding of (blackbox) neural rankers relative to (well-understood) traditional rankers, help understand the particular experimental setting of MS-Marco-based test collections.

show abstract

SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval

Cited by 40 publications

References 22 publications

Text and Code Embeddings by Contrastive Pre-Training

Text and Code Embeddings by Contrastive Pre-Training

Sparsifying Sparse Representations for Passage Retrieval by Top-$k$ Masking

How Different are Pre-trained Transformers for Text Ranking?

Contact Info

Product

Resources

About