On Aggregating Labels from Multiple Crowd Workers to Infer Relevance of Documents

Hosseini, Mehdi; Cox, Ingemar J.; Milić-Frayling, Nataša; Kazai, Gabriella; Vinay, V.

doi:10.1007/978-3-642-28997-2_16

Cited by 53 publications

(60 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, other sources of uncertainty could be considered. Recent research is particularly concerned with measuring uncertainty of the systems performance due to (i ) partial relevance judgments [2,4,20] and (ii ) errors in the relevance judgments made by human assessors [6,12]. Our future work will expand the theoretical model to incorporate additional sources of uncertainty and explore more general cost models for constructing test collections.…”

Section: Discussionmentioning

confidence: 99%

An uncertainty-aware query selection model for evaluation of IR systems

Hosseini

Cox

Milić-Frayling

et al. 2012

Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

We propose a mathematical framework for query selection as a mechanism for reducing the cost of constructing information retrieval test collections. In particular, our mathematical formulation explicitly models the uncertainty in the retrieval effectiveness metrics that is introduced by the absence of relevance judgments. Since the optimization problem is computationally intractable, we devise an adaptive query selection algorithm, referred to as Adaptive, that provides an approximate solution. Adaptive selects queries iteratively and assumes that no relevance judgments are available for the query under consideration. Once a query is selected, the associated relevance assessments are acquired and then used to aid the selection of subsequent queries. We demonstrate the effectiveness of the algorithm on two TREC test collections as well as a test collection of an online search engine with 1000 queries. Our experimental results show that the queries chosen by Adaptive produce reliable performance ranking of systems. The ranking is better correlated with the actual systems ranking than the rankings produced by queries that were selected using the considered baseline methods.

show abstract

Section: Discussionmentioning

confidence: 99%

An uncertainty-aware query selection model for evaluation of IR systems

Hosseini

Cox

Milić-Frayling

et al. 2012

Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

show abstract

“…So, we set the range of d as [0. 5,3] with a step size of 0.5. For the criterion c, the reported c suggests that both NIST and crowdsourced assessors are conservative with NIST assessors being more conservative than the crowdsourced workers [11,12].…”

Section: Experiments Settingsmentioning

confidence: 99%

“…It is well known that secondary assessors produce relevance judgments that differ from those that are or would be produced by primary assessors [4]. Whether there is a single secondary assessor, or a group of secondary assessors that are combined using sophisticated algorithms [5,6], there will be differences.…”

Section: Introductionmentioning

confidence: 99%

Tolerance of Effectiveness Measures to Relevance Judging Errors

Li¹,

Smucker

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract.Crowdsourcing relevance judgments for test collection construction is attractive because the practice has the possibility of being more affordable than hiring high quality assessors. A problem faced by all crowdsourced judgments -even judgments formed from the consensus of multiple workers -is that there will be differences in the judgments compared to the judgments produced by high quality assessors. For two TREC test collections, we simulated errors in sets of judgments and then measured the effect of these errors on effectiveness measures. We found that some measures appear to be more tolerant of errors than others. We also found that to achieve high rank correlation in the ranking of retrieval systems requires conservative judgments for average precision (AP) and nDCG, while precision at rank 10 requires neutral judging behavior. Conservative judging avoids mistakenly judging non-relevant documents as relevant at the cost of judging some relevant documents as non-relevant. In addition, we found that while conservative judging behavior maximizes rank correlation for AP and nDCG, to minimize the error in the measures' values requires more liberal behavior. Depending on the nature of a set of crowdsourced judgments, the judgments may be more suitable with some effectiveness measures than others, and the use of some effectiveness measures will require higher levels of judgment quality than others.

show abstract

“…Hauff et al compared all the above methods across 16 different test collections (from TREC and elsewhere), finding Soboroff et al's random-voting method best on nine collections, and Nuray and Can's Condorcet method best on six [8]. Hosseini et al used the EM framework to solve the problem of acquiring relevance judgements for Book Search tasks through crowdsourcing when no true relevance labels are available [9].…”

Section: Related Workmentioning

confidence: 99%

Reducing Reliance on Relevance Judgments for System Comparison by Using Expectation-Maximization

Gao

Webber

Oard

2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Relevance judgments are often the most expensive part of information retrieval evaluation, and techniques for comparing retrieval systems using fewer relevance judgments have received significant attention in recent years. This paper proposes a novel system comparison method using an expectationmaximization algorithm. In the expectation step, real-valued pseudo-judgments are estimated from a set of system results. In the maximization step, new system weights are learned from a combination of a limited number of actual human judgments and system pseudo-judgments for the other documents. The method can work without any human judgments, and is able to improve its accuracy by incrementally adding human judgments. Experiments using TREC Ad Hoc collections demonstrate strong correlations with system rankings using pooled human judgments, and comparison with existing baselines indicates that the new method achieves the same comparison reliability with fewer human judgments.

show abstract

On Aggregating Labels from Multiple Crowd Workers to Infer Relevance of Documents

Cited by 53 publications

References 16 publications

An uncertainty-aware query selection model for evaluation of IR systems

An uncertainty-aware query selection model for evaluation of IR systems

Tolerance of Effectiveness Measures to Relevance Judging Errors

Reducing Reliance on Relevance Judgments for System Comparison by Using Expectation-Maximization

Contact Info

Product

Resources

About