Scaling IR-system evaluation using term relevance sets

Abstract

This paper describes an evaluation method based on Term Relevance Sets (Trels) that measures an IR system's quality by examining the content of the retrieved results rather than by looking for pre-specified relevant pages. Trels consist of a list of terms believed to be relevant for a particular query as well as a list of irrelevant terms. The proposed method does not involve any document relevance judgments, and as such is not adversely affected by changes to the underlying collection. Therefore, it can better scale to very large, dynamic collections such as the Web. Moreover, this method can evaluate a system's effectiveness on an updatable "live" collection, or on collections derived from different data sources. Our experiments show that the proposed method is very highly correlated with official TREC measures.

Increasing cheat robustness of crowdsourcing tasks

“…A well-known example from the area of information retrieval is the series of extensive corpora created in the context of the NIST's Text REtrieval Conference (TREC) (Harman 1993). Since the manual creation of such resources typically requires substantial amounts of time and money there have long been advances into using automatically generated or extracted resources (Riloff 1996;Lesher and Sanelli 2000;Soboroff et al 2001;Amitay et al 2004). While there are several promising directions and methods that are reported to correlate well with human judgements, for many applications that require high precision, human judgements are still necessary (Marcus et al 1993).…”

Sampling search-engine results

“…A similar application is suggested by the paper [2] where the authors demonstrate how finding (by "hand") new terms relevant or irrelevant to a given query can be useful for building "corpus independent" performance measures for information retrieval systems. The main idea is that by providing a set of relevant and a set of irrelevant terms for a given query, we can evaluate the performance of the information retrieval system by checking whether the documents retrieved contained the specified relevant and irrelevant terms.…”

Emerging semantic communities in peer web search

“…Other techniques, such as the one proposed in [2], rely on users' assessments of term relevance to topics rather than document relevance to queries. Document-query relevance is automatically inferred based on the available term-topic relevance assessments.…”