The influence of scale form on relevance judgments

Katter, Robert V.

doi:10.1016/0020-0271(68)90002-8

Cited by 32 publications

(19 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…He found that the addition of a center anchor to a scale with two end categories resulted in increased reliability. Recall that Katter (1968) made a similar observation.…”

Section: Introductionmentioning

confidence: 66%

“…Similarly, Eisenberg and Hu (1987) labeled only the two end categories for their seven-point scale. Based on empirical results, Katter (1968) argued that providing a single-item anchor at the midpoint was not adequate and that "adding additional anchors at the extremes seems a possible solution" (p. 10). In the psychometric tradition, Bendig (1953) studied the effect on reliability of verbal anchoring.…”

Section: Introductionmentioning

confidence: 98%

“…For instance, Katter (1968) investigated type of scale in a study that compared ranking and category rating scales. He found that rankjudged data produce higher numbers of reversals, thereby leading to higher distortion in the true judgment value.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards the identification of the optimal number of relevance categories

Tang

Shaw

Vevea

1999

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

In this paper, we are concerned with participants' confidence in their judgments of the relevance of bibliographic records to particular research questions. We describe an empirical investigation of the association between judges' confidence and the number of categories for a relevance rating scale. Participants rated the relevance of bibliographic records, and recorded their confidence in the relevance ratings. We hypothesize that confidence in relevance judgments is a function of the number of relevance categories that are available in the rating scale. We consider scales ranging from 2 to 11 points, and define the optimal scale as the one for which participants express a maximum level of confidence. A pilot study finds no optimal number of points (because confidence continues to improve slightly through the 11‐point scale); nevertheless, the study shows little added benefit associated with scales that have more than six points. On the basis of the findings in that study, we adjusted our experimental procedures and found, in our principal study, that the optimal scale for maximizing confidence in relevance judgments has approximately seven points. We also present exploratory results involving gender effects, and the comparison of scales that have an odd number of points (for which a neutral judgment is possible) with scales that have an even number of points.

show abstract

“…He found that the addition of a center anchor to a scale with two end categories resulted in increased reliability. Recall that Katter (1968) made a similar observation.…”

Section: Introductionmentioning

confidence: 66%

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

Towards the identification of the optimal number of relevance categories

Tang

Shaw

Vevea

1999

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…Although it is difficult to ensure the same level of subject knowledge for all judges, the deviation can be much more with random volunteers than with evaluation organisers. It is indeed important to understand how this factor affects the reliability of an evaluation's results since it has been acknowledged in literature that the more knowledge and familiarity the judges have with the subject area, the less leniency they have for accepting documents as relevant (Rees and Schultz, 1967;Cuadra, 1967;Katter, 1968). Interestingly, Blanco et al (2013) analysed the impact of this factor on the reliability of the SemSearch evaluations and concluded that 1) experts are more pessimistic in their scoring and thus, accept fewer items as relevant when compared to workers (which agrees with the previous studies) and 2) crowdsourcing judgements, hence, cannot replace expert evaluations.…”

Section: Relevance Judgementsmentioning

confidence: 99%

An overview of semantic search evaluation initiatives

Elbedweihy

Wrigley

Clough

et al. 2015

Journal of Web Semantics

View full text Add to dashboard Cite

“…A number of authors attempted to introduce graded notions of relevance, either in terms of discrete categories [38,39,65], or in terms of points or confidence intervals on the real line [42,60]. Some authors also prefer to re-cast graded relevance as utility [28], or as a decision-theoretic notion of risk [153].…”

Section: The Binary Nature Of Relevancementioning

confidence: 99%

A Generative Theory of Relevance

Lavrenko¹

2009

The Information Retrieval Series

View full text Add to dashboard Cite

The influence of scale form on relevance judgments

Cited by 32 publications

References 3 publications

Towards the identification of the optimal number of relevance categories

Towards the identification of the optimal number of relevance categories

An overview of semantic search evaluation initiatives

A Generative Theory of Relevance

Contact Info

Product

Resources

About