Towards the identification of the optimal number of relevance categories

Abstract

In this paper, we are concerned with participants' confidence in their judgments of the relevance of bibliographic records to particular research questions. We describe an empirical investigation of the association between judges' confidence and the number of categories for a relevance rating scale. Participants rated the relevance of bibliographic records, and recorded their confidence in the relevance ratings. We hypothesize that confidence in relevance judgments is a function of the number of relevance categories that are available in the rating scale. We consider scales ranging from 2 to 11 points, and define the optimal scale as the one for which participants express a maximum level of confidence. A pilot study finds no optimal number of points (because confidence continues to improve slightly through the 11‐point scale); nevertheless, the study shows little added benefit associated with scales that have more than six points. On the basis of the findings in that study, we adjusted our experimental procedures and found, in our principal study, that the optimal scale for maximizing confidence in relevance judgments has approximately seven points. We also present exploratory results involving gender effects, and the comparison of scales that have an odd number of points (for which a neutral judgment is possible) with scales that have an even number of points.

Regions and levels: Measuring and mapping users' relevance judgments

“…Although Tang et al (1999) implied that a 7-point scale might provide users with the greatest confidence in making their relevance judgments, the analysis leading to that conclusion is flawed. The major problem is that the removal of data influences the frequency distribution of the relevance judgments on the various scales included in the study.…”

An overview of semantic search evaluation initiatives

“…Robertson (1981) argues that relevance should be treated as a continuous variable and hence, different levels of relevance should be incorporated in an evaluation model. Therefore, researchers have attempted to experiment with non-dichotomous relevance scales (Cuadra, 1967;Eisenberg, 1988;Janes, 1993;Spink et al, 1998;Tang et al, 1999). The study by Tang et al (1999) showed that a graded relevance scale with seven points led to the highest levels of confidence by the judges during their assessments.…”

“…Therefore, researchers have attempted to experiment with non-dichotomous relevance scales (Cuadra, 1967;Eisenberg, 1988;Janes, 1993;Spink et al, 1998;Tang et al, 1999). The study by Tang et al (1999) showed that a graded relevance scale with seven points led to the highest levels of confidence by the judges during their assessments. The additional benefit of using graded relevance scales is that a wider range of system effectiveness measures, such as Discounted Cumulative Gain (DCG), can be used (see Section 3.5).…”

Methods for Evaluating Interactive Information Retrieval Systems with Users

“…In IIR, a range of scales points are provided (although it is most common to have five-or seven-point scales), a range of scale labels are provided (e.g., not at all, somewhat, very much) and a summative measure is usually not produced, which is why this type of scaling is often referred to as Likert-type. There is little guidance about the appropriate number of scale points, although some researchers have suggested that sevenpoints are optimal for eliciting relevance assessments [265]. What is clear is that an odd number of scale points allow subjects to select a mid-point, while an even number does not.…”