An Improved Multileaving Algorithm for Online Ranker Evaluation

Abstract

Online ranker evaluation is a key challenge in information retrieval. An important task in the online evaluation of rankers is using implicit user feedback for inferring preferences between rankers. Interleaving methods have been found to be efficient and sensitive, i.e. they can quickly detect even small differences in quality. It has recently been shown that multileaving methods exhibit similar sensitivity but can be more efficient than interleaving methods. This paper presents empirical results demonstrating that existing multileaving methods either do not scale well with the number of rankers, or, more problematically, can produce results which substantially differ from evaluation measures like NDCG. The latter problem is caused by the fact that they do not correctly account for the similarities that can occur between rankers being multileaved. We propose a new multileaving method for handling this problem and demonstrate that it substantially outperforms existing methods, in some cases reducing errors by as much as 50%.

Sensitive and Scalable Online Evaluation with Theoretical Guarantees

“…To address this gap, we propose a novel multileaved comparison method, Pairwise Preference Multileaving (PPM). PPM di ers from existing multileaved comparison methods as its comparisons are based on inferred pairwise document preferences, whereas existing multileaved comparison methods either use some form of document assignment [27,28] or click credit functions [2,27]. We prove that PPM meets both the considerateness and the delity requirements, thus PPM guarantees correct winners in unambiguous cases while maintaining the user experience at all times.…”

“…Sample-Scored-Only Multileaving (SOSM) was introduced by Brost et al [2] in an a empt to create a more scalable multileaved comparison method. It is the only existing multileaved comparison method that does not have an interleaved comparison counterpart.…”

“…e metric by which multileaved comparison methods are compared is the binary error, E bin [2,27,28]. LetP nm be the preference inferred by a multileaved comparison method; then the error is:…”

“…While experiments using real users are preferred [4,6,18,31], most researchers do not have access to search engines. As a result the most common way of comparing online evaluation methods is by using simulated user behaviour [2,12,13,27,28]. Such simulated experiments show the performance of multileaved comparison methods when user behaviour adheres to a few simple assumptions.…”

“…For interleaved comparison, two experimental conditions ("control" and "treatment") are typical. Recently, multileaved comparisons have been introduced for the purpose of e ciently comparing large numbers of rankers [2,27]. ese multileaved comparison methods were introduced as an extension to interleaving and the majority are directly derived from their interleaving counterparts [27,28].…”