Learning to Rank in the Age of Muppets: Effectiveness–Efficiency Tradeoffs in Multi-Stage Ranking

Abstract

It is well known that rerankers built on pretrained transformer models such as BERT have dramatically improved retrieval effectiveness in many tasks. However, these gains have come at substantial costs in terms of efficiency, as noted by many researchers. In this work, we show that it is possible to retain the benefits of transformer-based rerankers in a multi-stage reranking pipeline by first using feature-based learning-to-rank techniques to reduce the number of candidate documents under consideration without adversely affecting their quality in terms of recall. Applied to the MS MARCO passage and document ranking tasks, we are able to achieve the same level of effectiveness, but with up to 18× increase in efficiency. Furthermore, our techniques are orthogonal to other methods focused on accelerating transformer inference, and thus can be combined for even greater efficiency gains. A higher-level message from our work is that, even though pretrained transformers dominate the modern IR landscape, there are still important roles for "traditional" LTR techniques, and that we should not forget history.1 Muppets being a whimsical way to refer to BERT and related transformer models.

Finding the Law: Enhancing Statutory Article Retrieval via Graph Neural Networks

“…Table 2 shows retrieval performance on BSARD test set. Although we report model performance on two rank-aware metrics (i.e., mAP and mRP), we emphasize that our approach is specifically aimed at improving the pre-fetching component of a retriever (Zhang et al, 2021a) and therefore focuses on optimizing rank-unaware metrics (i.e., R@k). First, we compare the performance of our proposed G-DSR model (8) against other well-known retrieval approaches and find it significantly outperforms all of them on SAR.…”

Injecting the BM25 Score as Text Improves BERT-Based Re-rankers

“…In this paper, we investigate multiple linear and non-linear interpolation ensemble methods to analyze the performance of them for combining BM25 and CE CAT scores in comparison to CE BM25CAT . For the sake of a fair analysis, we do not compare CE BM25CAT with a Learning-to-rank approach that is trained on 87 features by [65]. The use of ensemble methods brings additional overhead in terms of efficiency because it adds one more extra step to the re-ranking pipeline.…”

Sparse Pairwise Re-ranking with Pre-trained Transformers

“…Additionally, the theoretical analysis of 1 https://github.com/webis-de/ICTIR-22 duoBERT and duoT5 is still in its infancy; previous work even found such models difficult to be interpreted [27,41]. The effectiveness of duoBERT or duoT5 relies on computing preferences for all pairs of documents, at the expense of their efficiency [45] limiting their applicability in search scenarios with run time constraints.…”

“…The high computational cost of re-ranking documents with pre-trained transformers has recently received attention [20]. Even for pointwise approaches, the inference overhead can be prohibitive for practical applications [45]. There are two ideas to improve the efficiency of neural re-rankers: (1) improving the efficiency of the ranking model, and (2) reducing the required number of inferences.…”

“…Approaches to the former include early-exiting from inference by intermediate between-layer classification in BERT-like models [44], model distillation [18,19], or improved dense representations [39]. For the latter, Zhang et al [45] propose to introduce filtering steps in multi-stage re-ranking pipelines. They utilize feature-based learning to rank to compute a set of candidate documents that is then re-ranked using a BERT-like neural model, increasing efficiency by a factor of up to 18 compared to an unfiltered baseline at the same effectiveness.…”