Score adjustment for correction of pooling bias

Abstract

Information retrieval systems are evaluated against test collections of topics, documents, and assessments of which documents are relevant to which topics. Documents are chosen for relevance assessment by pooling runs from a set of existing systems. New systems can return unassessed documents, leading to an evaluation bias against them. In this paper, we propose to estimate the degree of bias against an unpooled system, and to adjust the system's score accordingly. Bias estimation can be done via leave-one-out experiments on the existing, pooled systems, but this requires the problematic assumption that the new system is similar to the existing ones. Instead, we propose that all systems, new and pooled, be fully assessed against a common set of topics, and the bias observed against the new system on the common topics be used to adjust scores on the existing topics. We demonstrate using resampling experiments on TREC test sets that our method leads to a marked reduction in error, even with only a relatively small number of common topics, and that the error decreases as the number of topics increases.

The Reusability of a Diversified Search Test Collection

“…Note that we are questioning the reusability of existing test collections: thus, "on-line" methods such as one that monitors the reusability of a test collection while building it [4] is outside the scope of this study. Also, while a score adjustment approach [29] may be useful for evaluating non-contributors accurately, this is also beyond our scope as it requires some new relevance assessments for the non-contributors.…”

Prioritizing relevance judgments to improve the construction of IR test collections

“…Weber and Park [1] estimate the bias that the uniform pooling and incomplete judgments introduce when un-judged documents are considered as non-relevant and when they are simply omitted from the computation of the performance scores. For each participating system they consider the discrepancy in a system's performance score when the pool first excludes and then includes documents uniquely retrieved by that system.…”

“…Weber and Park [1] partially address that issue by considering a more precise error estimation based on a set of common topics for which existing systems and a new one are fully assessed. By removing the uncertainty of the un-judged documents they propose an adjusted estimator that can be extrapolated to new topics and new systems.…”

“…Each row corresponds to a system and each column to a query 1 . An entry ‫ݔ‬ , in X denotes the performance score of the i-th system on the j-th query.…”

“…Particular attention is given to the work of Weber and Park [1], against which we compare our algorithm. Section 3 then provides a detailed description of our algorithm, while Section 4 describes specific implementation issues.…”