A similarity measure for indefinite rankings

Abstract

Ranked lists are encountered in research and daily life, and it is often of interest to compare these lists, even when they are incomplete or have only some members in common. An example is document rankings returned for the same query by different search engines. A measure of the similarity between incomplete rankings should handle non-conjointness, weight high ranks more heavily than low, and be monotonic with increasing depth of evaluation; but no measure satisfying all these criteria currently exists. In this article, we propose a new measure having these qualities, namely rank-biased overlap (RBO). The RBO measure is based on a simple probabilistic user model. It provides monotonicity by calculating, at a given depth of evaluation, a base score that is non-decreasing with additional evaluation, and a maximum score that is non-increasing. An extrapolated score can be calculated between these bounds if a point estimate is required. RBO has a parameter which determines the strength of the weighting to top ranks. We extend RBO to handle tied ranks and rankings of different lengths. Finally, we give examples of the use of the measure in comparing the results produced by public search engines, and in assessing retrieval systems in the laboratory.

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

“…To shed light on these considerations, Yilmaz et al [49] introduced an AP-based correlation coe cient called τ ap to achieve a top-weighted emphasis. Webber et al [48] also proposed Rank Biased Overlap (RBO) to operate over inde nite and non-conjoint rankings. Other evaluation criteria including evaluation based on judgment cost [7,8,35,49], coverage [38], inversions, interpretation and volatility matrix were also proposed in the past few years [33].…”

A functional genomic meta-analysis of clinical trials in systemic sclerosis: towards precision medicine and combination therapy

“…13, 2016; We determined if bottom-ranking genes in the fresolimumab base list were high-ranking genes in the MMF post list-that is, if genes most like those elevated in fresolimumab nonimprovers were also genes most like those down-regulated uniquely in MMF improvers. First, we compared the fresolimumab base and MMF post lists using a metric called rank biased overlap (RBO; the extrapolated version (Webber et al, 2010)), which is 1 if they are exactly the same, 0 if they are exactly opposite, and 0.5 if the rankings are random with respect to each other (Figure 4a). The fresolimumab base and MMF post ranked lists have RBO=0.34, suggesting that they are significantly dissimilar.…”

Investigating UI Displacements in an Adaptive Mobile Homescreen

“…We can also compare the distance between two lists using similarity measures. In their review of similarity measures, Webber et al (2010) identify Average Overlap (AO) as a weighted non-conjoint measure, that applies weight to higher ranked items by averaging over matches in list prefixes of length 1 -k. Kendall's tau is a non-weighted measure, and Fagin et al (2003) describe a way to transform this into a non-conjoint measurement, by considering a penalty for the case where items exist in one list but not the other. This penalty can be set to 0 to find the minimum distance, K (min), or 0.5 to find the average distance, K (avg).…”