Predicting relevance based on assessor disagreement: analysis and practical applications for search evaluation

Abstract

Evaluation of search engines relies on assessments of search results for selected test queries, from which we would ideally like to draw conclusions in terms of relevance of the results for general (e.g., future, unknown) users. In practice however, most evaluation scenarios only allow us to conclusively determine the relevance towards the particular assessor that provided the judgments. A factor that cannot be ignored when extending conclusions made from assessors towards users, is the possible disagreement on relevance, assuming that a single gold truth label does not exist. This paper presents and analyzes the Predicted Relevance Model (PRM), which allows predicting a particular result's relevance for a random user, based on an observed assessment and knowledge on the average disagreement between assessors. With the PRM, existing evaluation metrics designed to measure binary assessor relevance, can be transformed into more robust and effectively graded measures that evaluate relevance towards a random user. It also leads to a principled way of quantifying multiple graded or categorical relevance levels for use as gains in established graded relevance measures, such as normalized discounted cumulative gain (nDCG), which nowadays often use heuristic and data-independent gain values. Given a set of test topics with graded relevance judgments, the PRM allows evaluating systems on different scenarios, such as their capability of retrieving top results, or how well they are able to filter out non-relevant T. Demeester, C. Develder Ghent University -iMinds Belgium

Machine learning-based ABA treatment recommendation and personalization for autism spectrum disorder: an exploratory study

“… Cumulative Gain@k (CG@k) is the sum of the gains associated with the first k recommended items in any sequence. Gain is the score assigned to each recommended item based on its relevancy, and CG is the sum of all recommendation outcomes graded relevance scores [ 113 ]. The challenge with CG is that it ignores the result set’s rank when calculating its utility: Discounted Cumulative Gain@k (DCG@k) weighs each recommendation score based on its position.…”

How We Do Things With Words: Analyzing Text as Social and Cultural Data

“…Disagreement between annotators can signal weaknesses of the annotation scheme, or highlight the inherent ambiguity in what we are trying to measure. Disagreement itself can be meaningful and can be integrated in subsequent analyses (Aroyo and Welty, 2013 ; Demeester et al, 2016 ).…”

Information retrieval evaluation using test collections

“…The analysis also demonstrates that different effectiveness metrics lead to substantially different withinsystem variances, and therefore topic set size estimates, highlighting the importance of choosing metrics that are appropriate for the search task being analyzed. Demeester et al (2016) address the issue of the generalizability of relevance assessments and how this impacts the reliability of retrieval results in their paper, Predicting Relevance based on Assessor Disagreement: Analysis and Practical Applications for Search Evaluation. When test collections are built, it is common for one, or a small number of people to make relevance assessments of the information objects that have been returned for topics.…”

“…However, this is accepted as one of the limitations of test collection-based evaluation. Demeester et al (2016) introduce the Predicted Relevance Model (PRM), which predicts the relevance of a result for a random user, based on an observed assessment and knowledge of the average disagreement between assessors. The basic idea is that a greater degree of disagreement leads to a more uncertain prediction of relevance.…”