Towards Meaningful Statements in IR Evaluation: Mapping Evaluation Measures to Interval Scales

Abstract

Information Retrieval (IR) is a discipline deeply rooted in evaluation since its inception. Indeed, experimentally measuring and statistically validating the performance of IR systems are the only possible ways to compare systems and understand which are better than others and, ultimately, more effective and useful for end-users. Since the seminal paper by Stevens [103], it is known that the properties of the measurement scales determine the operations you should or should not perform with values from those scales. For example, Stevens suggested that you can compute means and variances only when you are working with, at least, interval scales. It was recently shown that the most popular evaluation measures in IR are not intervalscaled. However, so far, there has been little or no investigation in IR on the impact and consequences of departing from scale assumptions. Taken to the extremes, it might even mean that decades of experimental IR research used potentially improper methods, which may have produced results needing further validation. However, it was unclear if and to what extent these findings apply to actual evaluations; this opened a debate in the community with researchers standing on opposite positions about whether this should be considered an issue (or not) and to what extent. In this paper, we first give an introduction to the representational measurement theory explaining why certain operations and significance tests are permissible only with scales of a certain level. For that, we introduce the notion of meaningfulness specifying the conditions under which the truth (or falsity) of a statement is invariant under permissible transformations of a scale. Furthermore, we show how the recall base and the length of the run may make comparison and aggregation across topics problematic. Then we propose a straightforward and powerful approach for turning an evaluation measure into an interval scale, and describe an experimental evaluation of the differences between the original measures and the intervalscaled ones. For all the regarded measures -namely Precision, Recall, Average Precision, (Normalized) Discounted Cumulative Gain, Rank-Biased Precision and Reciprocal Rank -we observe substantial effects, both on the order of average values and on the outcome of significance tests. For the latter, previously significant differences turn out to be insignificant, while insignificant ones become significant. The effect varies remarkably between the tests considered but on average, we observed a 25% change in the decision about which systems are significantly different and which are not. These experimental findings further support the idea that measurement scales matter and that departing from their assumptions has an impact. This not only suggests that, to the extent possible, it would be better to comply with such assumptions but it also urges us to clearly indicate when we depart from such assumptions and, carefully, point out the limitations of the conclusions we draw and under which conditions they are drawn.

On Survivorship Bias in MS MARCO

“…Specifically, we find that across all four datasets, there are significant differences in performance when training with 𝑟 = 1 compared to the full dataset. Despite large absolute differences for RR on DL 19 and 20, the measure is statistically unstable [7] (especially with few queries available for TREC DL), and did not result in significance. In some cases, there is also statistically significant differences between 𝑟 = 5 and the full dataset (Dev RR@10, DL20 AP, and Robust04 nDCG@20).…”

“…Given a minimum rank 𝑟 , we filter down the official MS MARCO training triple sequence to only samples where the positive query-document pair appears within the top 𝑟 documents presented to the annotator. 7 Given the high effectiveness and ease of training of the monoT5 model, we select it as a representative ranking model for this experiment. Using both 𝑟 = 1 and 𝑟 = 5, we train a monoT5 over 256k samples, a mini-batch size of 8, and a learning rate of 5 × 10 −5 .…”

Shallow pooling for sparse labels

“…In this section, we provide only the background required to fully understand of the work reported in the current paper. In particular, Craswell et al [11] address concerns raised by Ferrante et al [12] who apply measurement theory to draw attention to important shortcomings of established evaluation measures, such as MRR. Many of these measures are not interval scaled, and therefore many common statistical tests are not permissible, and properly these measures should not even be averaged.…”

Learning-to-Rank at the Speed of Sampling: Plackett-Luce Gradient Estimation With Minimal Computational Complexity

“…All reported results are averages over 30 independent runs performed under identical circumstances. We report Normalized 𝐷𝐶𝐺@𝐾 (NDCG@K) as our ranking performance metric computed on the held-out test-set of each dataset; following the advice of Ferrante et al [9], we do not use querynormalization but dataset-normalization: we divide the 𝐷𝐶𝐺@𝐾 of a ranking model by the maximum possible 𝐷𝐶𝐺@𝐾 value on the entire test-set of the dataset.…”