Variations in relevance judgments and the evaluation of retrieval performance

Burgin, Robert

doi:10.1016/0306-4573(92)90031-t

Cited by 55 publications

(36 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The second response is a small set of studies that show that the comparative performance of two retrieval strategies is quite stable despite marked dierences in the relevance assessments themselves. Lesk and Salton (1969) and Burgin (1992) found no dierences in the relative performance of dierent indexing methods when evaluated using dierent sets of relevance judgments. Cleverdon (1970) found a few dierences in the ranks of 19 dierent indexing methods when evaluated using four independent sets of judgments, but the correlation between the methods' rankings was always very high and the absolute dierence in performance of the indexing methods was quite small.…”

Section: Introductionmentioning

confidence: 95%

Variations in relevance judgments and the measurement of retrieval effectiveness

Voorhees

1998

Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

221

View full text Add to dashboard Cite

Test collections have traditionally been used by information retrieval researchers to improve their retrieval strategies. To be viable as a laboratory tool, a collection must reliably rank dierent retrieval variants according to their true eectiveness. In particular, the relative eectiveness of two retrieval strategies should be insensitive to modest changes in the relevant document set since individual relevance assessments are known to vary widely.The test collections developed in the TREC workshops have become the collections of choice in the retrieval research community. To verify their reliability, NIST investigated the eect changes in the relevance assessments have on the evaluation of retrieval results. Very high correlations were found among the rankings of systems produced using dierent relevance judgment sets. The high correlations indicate that the comparative evaluation of retrieval performance is stable despite substantial dierences in relevance judgments, and thus rearm the use of the TREC collections as laboratory tools. Published by Elsevier Science Ltd.

show abstract

Section: Introductionmentioning

confidence: 95%

Variations in relevance judgments and the measurement of retrieval effectiveness

Voorhees

1998

Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

221

View full text Add to dashboard Cite

show abstract

“…However, they each cite two studies that do report empirical research (Cleverdon, 1970;Lesk & Salton, 1968a). Robert Burgin's ( 1992) article reports research findings, and also cites Kazhdan ( 1979) and Lesk & Salton ( 1968a). Finally, Lesk and Salton ( 1968a) report the results of their research and also cite Giulano and Jones ( 1966) and Rees et al ( 1967).…”

Section: Effects Of Variations In Relevance Assessments On Measures Omentioning

confidence: 81%

“…Finally, note that Lesk and Salton's research report (ISR-14) appearing in Figure 1 is the report on which their journal article is based. That is, Lesk & Salton ( 1968b) ( Burgin, 1992: Cleverdon, 1970Kazhdan, 1979;Lesk & Salton, 1968a). However.…”

Section: Effects Of Variations In Relevance Assessments On Measures Omentioning

confidence: 92%

See 1 more Smart Citation

Variations in relevance assessments and the measurement of retrieval effectiveness

Harter

1996

J. Am. Soc. Inf. Sci.

109

View full text Add to dashboard Cite

The purpose of this article is to bring attention to the problem of variations in relevance assessments and the effects that these may have on measures of retrieval effectiveness. Through an analytical review of the literature, I show that despite known wide variations in relevance assessments in experimental test collections, their effects on the measurement of retrieval performance are almost completely unstudied. I will further argue that what we know about the many variables that have been found to affect relevance assessments under experimental conditions, as well as our new understanding of psychological, situational, user-based relevance, point to a single conclusion. We can no longer rest the evaluation of information retrieval systems on the assumption that such variations do not significantly affect the measurement of information retrieval performance.A series of thorough, rigorous, and extensive tests is needed, of precisely how, and under what conditions, variations in relevance assessments do, and do not, affect measures of retrieval performance. We need to develop approaches to evaluation that are sensitive to these variations and to human factors and individual differences more generally. Our approaches to evaluation must reflect the real world of real users.

show abstract

“…He found that Lesk & Salton hypothesis is confirmed: the relative ranking of the seven different representations remained the same over two sets of judgments; however, there was one exception where ranking changed. Burgin (1992) used a collection of 1,239 documents in the cystic fibrosis collection (Shaw et al, 1991) that had one hundred queries with four sets of relevance judgments in the evaluation of six different document representations in order to compare performance as a function of different document representations and different judgment sets. The overall agreement between judgment sets was 40%.…”

Section: Tests Of Using Human Relevance Judgments In Ir Testsmentioning

confidence: 99%

Effects of Inconsistent Relevance Judgments on Information Retrieval Test Results: A Historical Perspective

Saračević¹

2008

lib

View full text Add to dashboard Cite

Only by continuous self-appraisal can a large information system make itself responsive to the needs of the scientific community. Concluding sentence in Lancaster (1969) AbstractThe main objective of information retrieval (IR) systems is to retrieve information or information objects relevant to user requests and possible needs. In IR tests, retrieval effectiveness is established by comparing IR systems retrievals (systems relevance) with users' or user surrogates' assessments (user relevance), where user relevance is treated as the gold standard for performance evaluation. Relevance is a human notion, and establishing relevance by humans is fraught with a number of problems-inconsistency in judgment being one of them. The aim of this critical review is to explore the relationship between relevance on the one hand and testing of IR systems and procedures on the other. Critics of IR tests raised the issue of validity of the IR tests because they were based on relevance judgments that are inconsistent. This review traces and synthesizes experimental studies dealing with (1) inconsistency of relevance judgments by people, (2) effects of such inconsistency on results of IR tests and (3) reasons for retrieval failures. A historical context for these studies and for IR testing is provided including an assessment of Lancaster's (1969) evaluation of MEDLARS and its unique place in the history of IR evaluation.

show abstract

Variations in relevance judgments and the evaluation of retrieval performance

Cited by 55 publications

References 20 publications

Variations in relevance judgments and the measurement of retrieval effectiveness

Variations in relevance judgments and the measurement of retrieval effectiveness

Variations in relevance assessments and the measurement of retrieval effectiveness

Effects of Inconsistent Relevance Judgments on Information Retrieval Test Results: A Historical Perspective

Contact Info

Product

Resources

About