Exploiting user disagreement for web search evaluation

Demeester, Thomas; Aly, Robin; Hiemstra, Djoerd; Nguyen, Dong; Trieschnigg, Dolf; Develder, Chris

doi:10.1145/2556195.2556268

Cited by 8 publications

(16 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To estimate the impact of this assumption, previous work studies the disagreement of assessors on binary labels and its influence on search engine comparisons (Voorhees 2001), leading to the conclusion that search engine comparisons are stable even under substantial assessor disagreement. Demeester et al (2014) show that in a graded relevance setting, this disagreement is especially strong on the top relevance levels. The current paper explicitly models the disagreement between assessors and particular scenarios of user relevance, such as users that are only satisfied with top results, or users that are looking for any result that is at least marginally relevant.…”

Section: Introductionmentioning

confidence: 86%

See 1 more Smart Citation

Predicting relevance based on assessor disagreement: analysis and practical applications for search evaluation

et al. 2015

Self Cite

View full text Add to dashboard Cite

Evaluation of search engines relies on assessments of search results for selected test queries, from which we would ideally like to draw conclusions in terms of relevance of the results for general (e.g., future, unknown) users. In practice however, most evaluation scenarios only allow us to conclusively determine the relevance towards the particular assessor that provided the judgments. A factor that cannot be ignored when extending conclusions made from assessors towards users, is the possible disagreement on relevance, assuming that a single gold truth label does not exist. This paper presents and analyzes the Predicted Relevance Model (PRM), which allows predicting a particular result's relevance for a random user, based on an observed assessment and knowledge on the average disagreement between assessors. With the PRM, existing evaluation metrics designed to measure binary assessor relevance, can be transformed into more robust and effectively graded measures that evaluate relevance towards a random user. It also leads to a principled way of quantifying multiple graded or categorical relevance levels for use as gains in established graded relevance measures, such as normalized discounted cumulative gain (nDCG), which nowadays often use heuristic and data-independent gain values. Given a set of test topics with graded relevance judgments, the PRM allows evaluating systems on different scenarios, such as their capability of retrieving top results, or how well they are able to filter out non-relevant T. Demeester, C. Develder Ghent University -iMinds Belgium

show abstract

Section: Introductionmentioning

confidence: 86%

“…Note that Demeester et al (2014) present evidence and a quantitative analysis of user disagreement, most of which will not be repeated here. For example, it was shown that for the FedWeb12 dataset (Nguyen et al 2012), the interassessor disagreement was much stronger than the intra-assessor disagreement.…”

Section: Introductionmentioning

confidence: 98%

Predicting relevance based on assessor disagreement: analysis and practical applications for search evaluation

et al. 2015

Self Cite

View full text Add to dashboard Cite

show abstract

“…where ∂q/∂Si,y and ∂F /∂Si,y can be obtained from Equation (4), (5) and (6). In order to take care of all the constraints of S, since the feasible set is the intersection of multiple linear inequalities, we can project the updated (possibly not feasible) Z into the feasible set after each update, by finding the closest point S in the feasible set Φν , which is a quadratic optimization problem minimizing the Frobenius norm ||S − Z||F with constraints.…”

Section: Now We Can Optimizementioning

confidence: 99%

“…Das et al [5] addressed the interactions of opinions between people connected by networks. Demeester et al [6] discussed the disagreement between different users on assessment of web search results. Their studies also focus on understanding annotator behavior, but none of them consider the case when multiple data items are organized into a batch.…”

Section: Related Workmentioning

confidence: 99%

Leveraging In-Batch Annotation Bias for Crowdsourced Active Learning

Zhuang

Young

2015

Proceedings of the Eighth ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

Data annotation bias is found in many situations. Often it can be ignored as just another component of the noise floor. However, it is especially prevalent in crowdsourcing tasks and must be actively managed. Annotation bias on single data items has been studied with regard to data difficulty, annotator bias, etc., while annotation bias on batches of multiple data items simultaneously presented to annotators has not been studied. In this paper, we verify the existence of "in-batch annotation bias" between data items in the same batch. We propose a factor graph based batch annotation model to quantitatively capture the in-batch annotation bias, and measure the bias during a crowdsourcing annotation process of inappropriate comments in LinkedIn. We discover that annotators tend to make polarized annotations for the entire batch of data items in our task. We further leverage the batch annotation model to propose a novel batch active learning algorithm. We test the algorithm on a real crowdsourcing platform and find that it outperforms in-batch bias naïve algorithms.

show abstract

“…Demeester et al . [8] discuss the disagreement between different users on assessment of web search results.…”

Section: Related Workmentioning

confidence: 99%

Debiasing Crowdsourced Batches

Zhuang

Parameswaran

Roth

2015

Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Crowdsourcing is the de-facto standard for gathering annotated data. While, in theory, data annotation tasks are assumed to be attempted by workers independently, in practice, data annotation tasks are often grouped into batches to be presented and annotated by workers together, in order to save on the time or cost overhead of providing instructions or necessary background. Thus, even though independence is usually assumed between annotations on data items within the same batch, in most cases, a worker's judgment on a data item can still be affected by other data items within the batch, leading to additional errors in collected labels. In this paper, we study the data annotation bias when data items are presented as batches to be judged by workers simultaneously. We propose a novel worker model to characterize the annotating behavior on data batches, and present how to train the worker model on annotation data sets. We also present a debiasing technique to remove the effect of such annotation bias from adversely affecting the accuracy of labels obtained. Our experimental results on both synthetic data and real-world data demonstrate the effectiveness of our proposed method.

show abstract

Exploiting user disagreement for web search evaluation

Cited by 8 publications

References 22 publications

Predicting relevance based on assessor disagreement: analysis and practical applications for search evaluation

Predicting relevance based on assessor disagreement: analysis and practical applications for search evaluation

Leveraging In-Batch Annotation Bias for Crowdsourced Active Learning

Debiasing Crowdsourced Batches

Contact Info

Product

Resources

About