Improving Bi-encoder Document Ranking Models with Two Rankers and Multi-teacher Distillation

Abstract

BERT-based Neural Ranking Models (NRMs) can be classified according to how the query and document are encoded through BERT's self-attention layers -bi-encoder versus cross-encoder. Biencoder models are highly efficient because all the documents can be pre-processed before the query time, but their performance is inferior compared to cross-encoder models. Both models utilize a ranker that receives BERT representations as the input and generates a relevance score as the output. In this work, we propose a method where multi-teacher distillation is applied to a crossencoder NRM and a bi-encoder NRM to produce a bi-encoder NRM with two rankers. The resulting student bi-encoder achieves an improved performance by simultaneously learning from a crossencoder teacher and a bi-encoder teacher and also by combining relevance scores from the two rankers. We call this method TRMD (Two Rankers and Multi-teacher Distillation). In the experiments, TwinBERT and ColBERT are considered as baseline bi-encoders. When monoBERT is used as the cross-encoder teacher, together with either TwinBERT or ColBERT as the bi-encoder teacher, TRMD produces a student bi-encoder that performs better than the corresponding baseline bi-encoder. For P@20, the maximum improvement was 11.4%, and the average improvement was 6.8%. As an additional experiment, we considered producing cross-encoder students with TRMD, and found that it could also improve the crossencoders. 1 CCS CONCEPTS• Information systems → Language models; • Computing methodologies → Neural networks.

Lightweight Composite Re-Ranking for Efficient Keyword Search with BERT

“…While BERT and CONV-KNRM also perform decently well, ColBERT doesn't perform as good as the other models. Our numbers for ColBERT are consistent with another recent study [5] that also trained ColBERT on Robust04 and ClueWeb09-Cat-B.…”

“…Following previous work [34], for Robust04 and ClueWeb09-Cat-B, we re-rank the 150 documents per query retrieved by Indri initial ranking in the Lemur system 5 . For the MS MARCO Dev set, we follow a common practice to re-rank top 1000 passages per query retrieved by BM25.…”

ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval

“…Another popular choice is ColBERT (Khattab and Zaharia, 2020), whose structure is more similar to dual-encoders, and thus allows KD on in-batch negative examples . Besides, a handful of studies also try to improve the performance with multi-teacher distillation (Choi et al, 2021;Hofstätter et al, 2021). However, none of them investigate how to more effectively distill the knowledge of teachers into a student with different architecture.…”

“…Cross-encoder. Researchers have developed passage re-ranking models (i.e., re-rankers) further to improve end-to-end QA after the retrieval of candidate passages (Choi et al, 2021;Ren et al, 2021b;. Using a cross-encoder as a re-ranker usually achieves superior performance.…”

“…It allows distillation on in-batch negative examples , which is critical for training dual-encoders (Karpukhin et al, 2020). Besides, there are also a handful of studies that adopt multi-teacher distillation, i.e., using cross-encoders and ColBERT simultaneously as the teachers (Choi et al, 2021;Hofstätter et al, 2021). Notably, all these studies have verified that it is fruitful to improve dual-encoders with such cross-architecture distillation setting, where the teacher is equipped with more expressive query-passage interactions than the dual-encoder student.…”