Low-cost and robust evaluation of information retrieval systems

Abstract

Research in Information Retrieval has progressed against a background of rapidly increasing corpus size and heterogeneity, with every advance in technology quickly followed by a desire to organize and search more unstructured, more heterogeneous, and even bigger corpora. But as retrieval problems get larger and more complicated, evaluating the ranking performance of a retrieval engine gets harder: evaluation requires human judgments of the relevance of documents to queries, and for very large corpora the cost of acquiring these judgments may be insurmountable. This cost limits the types of problems researchers can study as well as the data they can be studied on. We present methods for understanding performance differences between retrieval engines in the presence of missing and noisy relevance judgments. The work introduces a model of the cost of experimentation that incorporates the cost of human judgments as well as the cost of drawing incorrect conclusions about differences between engines in both the training and testing phases of engine development. Through adopting a view of evaluation that is more concerned with distributions over performance differences rather than estimates of absolute performance, the expected cost can be minimized so as to reliably differentiate between engines with less than 1% of the human effort that has been used in past experiments.

Measuring the reusability of test collections

“…Carterette showed that the mean and variance for precision at k and average precision have analytical forms [6]. Given a query Q ∈ Q, these analytical forms are:…”

“…Although we primarily focus on precision at k and average precision in this paper, it should be noted that analytical forms for the means and variances of other retrieval metrics exist, including recall and NDCG [6]. Thus, our intervalbased reusability measures can be easily applied to these metrics, as well.…”

Pseudo test collections for learning web search ranking functions

“…This has led researchers to pursue low cost strategies for constructing manual test collections. Two emerging evaluation paradigms are minimal test collections [8,7,6] and crowdsourcing [2]. Both of these strategies are useful for low-cost one-time evaluations.…”

“…The queries are either sampled from query logs or manually generated. Each query is then issued to one or more retrieval systems, which returns candidate documents that are then judged, either via pooling [21,37], the minimal test collection paradigm [8,7,6], or crowdsourcing [2].…”

Cross-corpus relevance projection

“…However, in more constrained research environments these options are not available, and relevance judgments are usually provided by humans. To reduce the cost of this potentially expensive process, researchers have developed low-cost evaluation strategies, including minimal test collections [2] and crowdsourcing [1]. Despite the usefulness of these strategies, the resulting relevance judgments cannot easily be "ported" to a new or different corpus.…”