Tackling Biased Baselines in the Risk-Sensitive Evaluation of Retrieval Systems

Abstract

Abstract. The aim of optimising information retrieval (IR) systems using a risksensitive evaluation methodology is to minimise the risk of performing any particular topic less effectively than a given baseline system. Baseline systems in this context determine the reference effectiveness for topics, relative to which the effectiveness of a given IR system in minimising the risk will be measured. However, the comparative risk-sensitive evaluation of a set of diverse IR systems -as attempted by the TREC 2013 Web track -is challenging, as the different systems under evaluation may be based upon a variety of different (base) retrieval models, such as learning to rank or language models. Hence, a question arises about how to properly measure the risk exhibited by each system. In this paper, we argue that no model of information retrieval alone is representative enough in this respect to be a true reference for the models available in the current state-of-the-art, and demonstrate, using the TREC 2012 Web track data, that as the baseline system changes, the resulting risk-based ranking of the systems changes significantly. Instead of using a particular system's effectiveness as the reference effectiveness for topics, we propose several remedies including the use of mean within-topic system effectiveness as a baseline, which is shown to enable unbiased measurements of the risk-sensitive effectiveness of IR systems.

Risk-Sensitive Evaluation and Learning to Rank using Multiple Baselines

“…1 Although our analysis is equally applicable to the TREC 2013 Web track, due to the lack of space, we report results from TREC 2012, which are also directly comparable to that of previous works [13,14]. Table 6 (where c = 50), T Risk > ±2 indicates that the observed U Risk score exhibits a significant level of risk.…”

“…In the subsequent year of the Web track [12], a standard baseline called indriC-ASP and based on the Indri retrieval platform was provided. Similar to [13,14], we use the same indriCASP system as the nominal single baseline on the 2012 Web track topics.…”

“…On the other hand, the comparative risk-sensitive evaluation of different IR systems is challenging, as the systems may be based upon a variety of different (base) retrieval models -such as learning to rank or language models -or upon different IR platforms (Indri, Terrier etc.). It has been shown that using a particular system as the baseline in a comparative risk-sensitive evaluation of a set of diverse IR systems -as attempted by the TREC 2013 and 2014 Web track -yields biased risk-reward tradeoff measurements [14], especially when the systems under evaluation are not variations of the provided baseline system. To address this, the use of the within-topic mean system performance was proposed as an unbiased baseline (as well as the within-topic median system performance and the within-topic maximum system performance).…”

“…, r. Since the arithmetic mean gives equal weight to every retrieval strategy in determining the within-topic mean system performance, a baseline system that is determined by the Meanq scores will be unbiased with respect to the retrieval strategies yielding the r system scores. However, as shown in [14], the use of Meanq exposes a problem about the validity of the comparative risk-sensitive evaluation of different IR systems. This issue is related to the risk-based rankings of the systems obtained using Meanq.…”

“…Indeed, such a comparison of the risk-sensitive performances of different IR systems actually implies the comparison of the retrieval effectiveness of the individual systems based on the underlying effectiveness measure, i.e. ERR-@20 [14]. That is, the ranking of the systems obtained by using the underlying effectiveness measure will be the same as the risk-based ranking of the systems obtained using the unbiased baseline Meanq, irrespective of the value of the risk sensitivity parameter α.…”