A White Box Analysis of ColBERT

Formal, Thibault; Piwowarski, Benjamin; Clinchant, Stéphane

doi:10.1007/978-3-030-72240-1_23

Cited by 23 publications

(17 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, Formal et al [12] showed that the dot product 𝜙 𝑇 𝑞 𝑖 𝜙 𝑑 𝑗 used by ColBERT implicitly encapsulates token importance, by giving higher scores to tokens that have higher IDF values.…”

Section: Multi Representation Dense Retrievalmentioning

confidence: 99%

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

Wang

Macdonald

Tonellotto

et al. 2021

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

Pseudo-relevance feedback mechanisms, from Rocchio to the relevance models, have shown the usefulness of expanding and reweighting the users' initial queries using information occurring in an initial set of retrieved documents, known as the pseudo-relevant set. Recently, dense retrieval -through the use of neural contextual language models such as BERT for analysing the documents' and queries' contents and computing their relevance scores -has shown a promising performance on several information retrieval tasks still relying on the traditional inverted index for identifying documents relevant to a query. Two different dense retrieval families have emerged: the use of single embedded representations for each passage and query (e.g. using BERT's [CLS] token), or via multiple representations (e.g. using an embedding for each token of the query and document). In this work, we conduct the first study into the potential for multiple representation dense retrieval to be enhanced using pseudo-relevance feedback. In particular, based on the pseudo-relevant set of documents identified using a first-pass dense retrieval, we extract representative feedback embeddings (using KMeans clustering) -while ensuring that these embeddings discriminate among passages (based on IDF) -which are then added to the query representation. These additional feedback embeddings are shown to both enhance the effectiveness of a reranking as well as an additional dense retrieval operation. Indeed, experiments on the MSMARCO passage ranking dataset show that MAP can be improved by upto 26% on the TREC 2019 query set and 10% on the TREC 2020 query set by the application of our proposed ColBERT-PRF method on a ColBERT dense retrieval approach.

show abstract

Section: Multi Representation Dense Retrievalmentioning

confidence: 99%

Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval

Wang

Macdonald

Tonellotto

et al. 2021

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Previous works scrutinizing BERT-based ranking models either relied on axiomatic approaches adapted to neural models [1,17], controlled experiments [11], or direct investigation of the learned representations [9,7] or attention [19]. This line of work has shown -among other findings -that these models, which rely on contextualized semantic matching, are actually still quite sensitive to lexical match and term statistics in documents/collections [9,7]. However, these observations are based on specifically tailored approaches that cannot directly be applied to any given model.…”

Section: Introductionmentioning

confidence: 99%

Match Your Words! A Study of Lexical Matching in Neural Information Retrieval

Formal¹,

Piwowarski²,

Clinchant³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Neural Information Retrieval models hold the promise to replace lexical matching models, e.g. BM25, in modern search engines. While their capabilities have fully shone on in-domain datasets like MS MARCO, they have recently been challenged on out-of-domain zero-shot settings (BEIR benchmark), questioning their actual generalization capabilities compared to bag-of-words approaches. Particularly, we wonder if these shortcomings could (partly) be the consequence of the inability of neural IR models to perform lexical matching off-the-shelf. In this work, we propose a measure of discrepancy between the lexical matching performed by any (neural) model and an "ideal" one. Based on this, we study the behavior of different state-of-the-art neural IR models, focusing on whether they are able to perform lexical matching when it's actually useful, i.e. for important terms. Overall, we show that neural IR models fail to properly generalize term importance on out-of-domain collections or terms almost unseen during training.

show abstract

“…Similarly, we postulate that the most important query tokens 3 are more likely to bring relevant documents than non-relevant documents, and therefore we propose to prune (remove) the unimportant query embeddings. Indeed, Formal et al [3] noted that exact matches and the more important terms contribute more to the overall ColBERT scores; we argue that these terms are those that should be the focus of the ANN search.…”

Section: Query Embedding Pruningmentioning

confidence: 62%