Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval 2010
DOI: 10.1145/1835449.1835526
|View full text |Cite
|
Sign up to set email alerts
|

Collecting high quality overlapping labels at low cost

Abstract: This paper studies quality of human labels used to train search engines' rankers. Our specific focus is performance improvements obtained by using overlapping relevance labels, which collecting multiple human judgments for each training sample. The paper explores whether, when, and for which should obtain overlapping training labels, as well as labels per sample are needed. The proposed scheme collects additional labels only for a subset of training samples, specifically for those that are labeled relevant by … Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
2
1

Citation Types

0
15
0

Year Published

2011
2011
2024
2024

Publication Types

Select...
5
2

Relationship

0
7

Authors

Journals

citations
Cited by 24 publications
(15 citation statements)
references
References 8 publications
0
15
0
Order By: Relevance
“…A commonly used strategy is simple Majority Voting (MV) [5,6]. In the MV method, the annotation that receives the maximum number of votes is treated as the final aggregated label, with ties broken randomly.…”
Section: Related Workmentioning
confidence: 99%
“…A commonly used strategy is simple Majority Voting (MV) [5,6]. In the MV method, the annotation that receives the maximum number of votes is treated as the final aggregated label, with ties broken randomly.…”
Section: Related Workmentioning
confidence: 99%
“…How should we compute consensus with such multi-labeling? For learning to rank [1,7], how sensitive is the learner to different quanitities and distributions of label noise?…”
Section: Introductionmentioning
confidence: 99%
“…Prior work by Yang et al [7] studied learning to rank (with graded judgments) rather than binary classification, evaluating SL, MV, and other consensus algorithms for ranking with LambdaRank. They assumed labels come from reliable experts and provided limited analysis of the relationship between consensus method and the resulting learning curve.…”
Section: Introductionmentioning
confidence: 99%
See 2 more Smart Citations