Efficient Document Re-Ranking for Transformers by Precomputing Term Representations

MacAvaney, Sean; Nardini, Franco Maria; Perego, Raffaele; Tonellotto, Nicola; Goharian, Nazli; Frieder, Ophir

doi:10.1145/3397271.3401093

Cited by 85 publications

(67 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evaluation on the Robust04 and GOV2 test collections confirms that BERT-QE significantly outperforms BERT-Large with relatively small extra computational cost (up to 30%). In future work, we plan to further im-prove the efficiency of BERT-QE, by combining the proposed BERT-QE with the pre-computation techniques proposed recently (Khattab and Zaharia, 2020;MacAvaney et al, 2020a), wherein most of the computations could be performed offline. There are two hyper-parameters in BERT-QE, namely α and β, both of which are interpolation coefficients.…”

Section: Discussionmentioning

confidence: 99%

BERT-QE: Contextualized Query Expansion for Document Re-ranking

Zheng¹,

Hui²,

He³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Query expansion aims to mitigate the mismatch between the language used in a query and in a document. However, query expansion methods can suffer from introducing non-relevant information when expanding the query. To bridge this gap, inspired by recent advances in applying contextualized models like BERT to the document retrieval task, this paper proposes a novel query expansion model that leverages the strength of the BERT model to select relevant document chunks for expansion. In evaluation on the standard TREC Robust04 and GOV2 test collections, the proposed BERT-QE model significantly outperforms BERT-Large models.

show abstract

Section: Discussionmentioning

confidence: 99%

BERT-QE: Contextualized Query Expansion for Document Re-ranking

Zheng¹,

Hui²,

He³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

show abstract

“…Lighter deep LM rankers are developed (MacAvaney et al, 2020;Gao et al, 2020), but their cross attention operations are still too expensive for fullcollection retrieval.…”

Section: Related Workmentioning

confidence: 99%

COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List

Gao¹,

Dai²,

Callan³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Classical information retrieval systems such as BM25 rely on exact lexical match and carry out search efficiently with inverted list index. Recent neural IR models shifts towards soft semantic matching all query document terms, but they lose the computation efficiency of exact match systems. This paper presents COIL, a contextualized exact match retrieval architecture that brings semantic lexical matching. COIL scoring is based on overlapping query document tokens' contextualized representations. The new architecture stores contextualized token representations in inverted lists, bringing together the efficiency of exact match and the representation power of deep language models. Our experimental results show COIL outperforms classical lexical retrievers and state-of-the-art deep LM retrievers with similar or smaller latency. 1

show abstract

“…Each module is a stack of transformer layers (Vaswani et al, 2017), initialized with weights from BERT. In a related approach, MacAvaney et al (2020) investigate the relationship between different numbers of dedicated layers of BERT for query-document interactions and measure the resulting speedup that is due to token representation caching, as well as its impact on the end-to-end ranking quality. Khattab and Zaharia (2020) propose a related approach, namely ColBERT.…”

Section: Key Concepts Of Neural Rankingmentioning

confidence: 99%

CoRT: Complementary Rankings from Transformers

Wrzalik¹,

Krechel²

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Many recent approaches towards neural information retrieval mitigate their computational costs by using a multi-stage ranking pipeline. In the first stage, a number of potentially relevant candidates are retrieved using an efficient retrieval model such as BM25. Although BM25 has proven decent performance as a first-stage ranker, it tends to miss relevant passages. In this context we propose CoRT, a simple neural first-stage ranking model that leverages contextual representations from pretrained language models such as BERT to complement term-based ranking functions while causing no significant delay at query time. Using the MS MARCO dataset, we show that CoRT significantly increases the candidate recall by complementing BM25 with missing candidates. Consequently, we find subsequent re-rankers achieve superior results with less candidates. We further demonstrate that passage retrieval using CoRT can be realized with surprisingly low latencies.

show abstract

Efficient Document Re-Ranking for Transformers by Precomputing Term Representations

Cited by 85 publications

References 19 publications

BERT-QE: Contextualized Query Expansion for Document Re-ranking

BERT-QE: Contextualized Query Expansion for Document Re-ranking

COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List

CoRT: Complementary Rankings from Transformers

Contact Info

Product

Resources

About