From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective

Abstract

Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -a sparse expansion-based retriever -and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained Language Model initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, leading to state-of-the-art results in both scenarios for sufficiently expressive models. CCS CONCEPTS• Information systems → Retrieval models and ranking.

UnifieR: A Unified Retriever for Large-Scale Retrieval

“…On the other hand, lexicon-based retrieval models, a.k.a. neural weighting schemes in sparse retrieval methods, are proposed to exploit intrinsic properties of natural language for sparse retrieval [43,42,13,12,14]. Recently, built upon causal language models (CLM) [46,47], [42] proposes to leverage the concurrence between a document and a query for lexicon-based sparse representation expansion.…”

“…In addition, we present a uni-retrieval scheme for fast yet effective large-scale retrieval. Instead of adding their scores [28,14] from twice-retrieval with heavy overheads, we pipelinelize the retrieval procedure: given q, our lexicon-based retrieval under an inverted file system is to retrieve top-K documents from D. Then, our dense-vector retrieval is then applied to the constrained candidates for dense scores. The final retrieval results are according to simple addition of the two scores.…”

“…As stated in §3.3, we take a 2-stage learning scheme [16]. We use coCondenser-marco [16] (unsupervised continual pre-training from BERT-base [9]) as our initialization as it shares a similar neural structure (see the end of §3.2) and has potential for promising performance [16,14,66]. θ (ctx) , θ (den) , and θ (lex) correspond to Transformer layers of 6, 6, and 2, respectively, where max length is 128 and warmup ratio is 5%.…”

“…'Passage+' denotes positive passage of the corresponding query. 'Indep-den' denotes a well-trained state-of-the-art dense-vector retrieval model with static hard negatives (i.e., coCondenser [16], M@10=38.2) while 'Indep-lex' denotes a well-trained stateof-the-art lexicon-based retrieval model with static hard negatives (i.e., SPLADE [14], M@10=38.5).…”

“…Nonetheless, such surface interactions -score aggregations [28], direct co-training [18], and logits distillations [5] -cannot fully exploit the benefits of the two paradigms -regardless of their complementary contextual features and distinct representation views. Specifically, as for contextual features, the dense-vector models focus more on sequence-level global embeddings against information bottleneck [37,15,16], whereas the lexicon-based models focus on word-level local contextual embeddings for precise lexicon-weighting [12,14,42]. Aligning the two retrieval paradigms more closely is likely to benefit each other since global-local contexts are proven complementary in general representation learning [53,1].…”