Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking

Santhanam, Keshav; Saad-Falcon, Jon; Franz, Martin; Khattab, Omar; Sil, Avirup; Florian, Radu; Sultan, Md Arafat; Roukos, Salim; Zaharia, Matei; Potts, Christopher

doi:10.48550/arxiv.2212.01340

Cited by 1 publication

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors show this by conducting a post-hoc comparison of published works as well as an in-depth cost analysis of representative methods (BM25, Dense Passage Retrieval, SPLADE, and ColBERTv2) to arrive at conclusions that are broadly consistent with the observations around model inference of Scells et al (2022). Santhanam et al (2022b) use this fact to encourage the adoption of multidimensional leaderboards and motivate research on metrics that capture the overall utility of a retrieval or ranking method in a single quantity. They point to the Dynascores proposed by Ma et al (2021b) as one such measure that allows for a single ranking of a collection of systems.…”

Section: Designing Multidimensional Leaderboardsmentioning

confidence: 89%

“…We too encourage the development of multidimensional leaderboards to incentivize research into efficient and effective systems. In fact, while Santhanam et al (2022b) argue for leaderboards that capture inference efficiency, we believe training efficiency too must be reflected in the overall utility of a retrieval and ranking system. In spite of arguments that training a model incurs a cost that is amortized and thus comparably insignificant, we note that retrieval and ranking models have a relatively short lifetime: As the data distribution shifts, models must often be re-trained or fine-tuned on fresh samples.…”

Section: Designing Multidimensional Leaderboardsmentioning

confidence: 96%

“…These leaderboards have demonstrably contributed to the progress we have witnessed over the years: MRR@10 for the MS MARCO passage retrieval task, as a representative example, has remarkably gained over 24 points since its debut! But as Santhanam et al (2022b) argue, the emphasis on quality hides the fact that some ranked lists are much more expensive to obtain than others. The authors show this by conducting a post-hoc comparison of published works as well as an in-depth cost analysis of representative methods (BM25, Dense Passage Retrieval, SPLADE, and ColBERTv2) to arrive at conclusions that are broadly consistent with the observations around model inference of Scells et al (2022).…”

Section: Designing Multidimensional Leaderboardsmentioning

confidence: 99%

See 2 more Smart Citations

ReNeuIR: Reaching Efficiency in Neural Information Retrieval

Bruch¹,

Lucchese

Nardini

2022

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

As information retrieval researchers, we not only develop algorithmic solutions to hard problems, but we also insist on a proper, multifaceted evaluation of ideas. The literature on the fundamental topic of retrieval and ranking, for instance, has a rich history of studying the effectiveness of indexes, retrieval algorithms, and complex machine learning rankers, while at the same time quantifying their computational costs, from creation and training to application and inference. This is evidenced, for example, by more than a decade of research on efficient training and inference of large decision forest models in Learning to Rank (LtR). As we move towards even more complex, deep learning models in a wide range of applications, questions on efficiency have once again resurfaced with renewed urgency. Indeed, efficiency is no longer limited to time and space; instead it has found new, challenging dimensions that stretch to resource-, sampleand energy-efficiency with ramifications for researchers, users, and the environment.This monograph takes a step towards promoting the study of efficiency in the era of neural information retrieval by offering a comprehensive survey of the literature on efficiency and effectiveness in ranking, and to a limited extent, retrieval. This monograph was inspired by the parallels that exist between the challenges in neural network-based ranking solutions and their predecessors, decision forest-based LtR models, as well as the connections between the solutions the literature to date has to offer. We believe that by understanding the fundamentals underpinning these algorithmic and data structure solutions for containing the contentious relationship between efficiency and effectiveness, one can better identify future directions and more efficiently determine the merits of ideas. We also present what we believe to be important research directions in the forefront of efficiency and effectiveness in retrieval and ranking.

show abstract

Section: Designing Multidimensional Leaderboardsmentioning

confidence: 89%

Section: Designing Multidimensional Leaderboardsmentioning

confidence: 96%