Variations in relevance assessments and the measurement of retrieval effectiveness

Abstract

The purpose of this article is to bring attention to the problem of variations in relevance assessments and the effects that these may have on measures of retrieval effectiveness. Through an analytical review of the literature, I show that despite known wide variations in relevance assessments in experimental test collections, their effects on the measurement of retrieval performance are almost completely unstudied. I will further argue that what we know about the many variables that have been found to affect relevance assessments under experimental conditions, as well as our new understanding of psychological, situational, user-based relevance, point to a single conclusion. We can no longer rest the evaluation of information retrieval systems on the assumption that such variations do not significantly affect the measurement of information retrieval performance.A series of thorough, rigorous, and extensive tests is needed, of precisely how, and under what conditions, variations in relevance assessments do, and do not, affect measures of retrieval performance. We need to develop approaches to evaluation that are sensitive to these variations and to human factors and individual differences more generally. Our approaches to evaluation must reflect the real world of real users.

Variations in relevance judgments and the measurement of retrieval effectiveness

“…The ®eld has almost as long a history of criticism of this experimental paradigm (Cuadra & Katter, 1967;Harter, 1996;Taube, 1965). The gist of the critics' complaint is that relevance is inherently subjective.…”

Evaluating Retrieval Effectiveness by Sustainable Rank List

“…For example, the degree of URL u 1 in Table 5 is 5; we normalized it by dividing with 17, which is the sum of all URLs degrees. Equation (6) shows the Pseudo Relevance score (PR s ) for the pages fetched for the IoT-related information need.…”

Creative professional users’ musical relevance criteria

“…Schamber arranges 80 such criteria in a table [7:11]. Harter [12] also comments on the wide range of criteria derived from such studies. Saracevic summarises these criteria, which he calls 'clues' [5:2130], noting that although there is variety in their labels, they are 'remarkably similar' in concept.…”