Semi-parametric analysis of multi-rater data

Rogers, Simon; Girolami, Mark; Polajnar, Tamara

doi:10.1007/s11222-009-9125-z

Cited by 12 publications

(14 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Model-based gold-standard estimation such as (Dawid and Skene, 1979) has long been the standard in epidemiology, and has been applied to disease prevalence estimation (Albert and Dodd, 2008) and also to many other problems such as human annotation of craters in images of Venus (Smyth et al, 1995). Smyth et al (1995), Rogers et al (2010), andRaykar et al (2010) all discuss the advantages of learning and evaluation with probabilistically annotated corpora. Rzhetsky et al (2009) and Whitehill et al ( 2009) estimate annotation models without gold-standard supervision, but neither models annotator biases, which are critical for estimating true labels.…”

Section: Related Workmentioning

confidence: 99%

The Benefits of a Model of Annotation

Passonneau

Carpenter

2014

TACL

114

View full text Add to dashboard Cite

Standard agreement measures for interannotator reliability are neither necessary nor sufficient to ensure a high quality corpus. In a case study of word sense annotation, conventional methods for evaluating labels from trained annotators are contrasted with a probabilistic annotation model applied to crowdsourced data. The annotation model provides far more information, including a certainty measure for each gold standard label; the crowdsourced data was collected at less than half the cost of the conventional approach.

show abstract

Section: Related Workmentioning

confidence: 99%

The Benefits of a Model of Annotation

Passonneau

Carpenter

2014

TACL

114

View full text Add to dashboard Cite

show abstract

“…Application areas include disease prevalence estimation (Albert and Dodd, 2008), identification of craters in images of Venus (Smyth et al, 1995), curation of biological data (Rzhetsky et al, 2009), computer vision (Whitehill et al, 2009), patient history (Dawid and Skene, 1979), and clinical reports (2010). Smyth et al (1995), Rogers et al, and (2010) and Raykar et al (2010) discuss the advantages of probabilistically annotated corpora over majority vote. Much of this work is motivated by the observation that annotators have different accuracies, and the fact that when annotators have known accuracies it can be shown that a majority of inaccurate annotators can be wrong (Raykar et al, 2010;Passonneau and Carpenter, 2014).…”

Section: Related Workmentioning

confidence: 99%

Estimation of Discourse Segmentation Labels from Crowd Data

Huang

Zhong

Passonneau³

2015

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

For annotation tasks involving independent judgments, probabilistic models have been used to infer ground truth labels from data where a crowd of many annotators labels the same items. Such models have been shown to produce results superior to taking the majority vote, but have not been applied to sequential data. We present two methods to infer ground truth labels from sequential annotations where we assume judgments are not independent, based on the observation that an annotator's segments all tend to be several utterances long. The data consists of crowd labels for annotation of discourse segment boundaries. The new methods extend Hidden Markov Models to relax the independence assumption. The two methods are distinct, so positive labels proposed by both are taken to be ground truth. In addition, results of the models are checked using metrics that test whether an annotator's accuracy relative to a given model remains consistent across different conversations.

show abstract

“…In the database and data mining research communities, various models have been proposed [17,21]. According to applications, they are used for discrete labeling [1, 6, 7, 11-13, 16, 20, 26, 32, 33, 39, 41, 42], or continuous labeling [13,18,26,30].…”

Section: Introductionmentioning

confidence: 99%

A Probabilistic Graphical Model for Analyzing the Subjective Visual Quality Assessment Data from Crowdsourcing

Ling²,

Wang

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

The swift development of the multimedia technology has raised dramatically the users' expectation on the quality of experience. To obtain the ground-truth perceptual quality for model training, subjective assessment is necessary. Crowdsourcing platform provides us a convenient and feasible way to run large-scale experiments. However, the obtained perceptual quality labels are generally noisy. In this paper, we propose a probabilistic graphical annotation model to infer the underlying ground truth and discovering the annotator's behavior. In the proposed model, the ground truth quality label is considered following a categorical distribution rather than a unique number, i.e., different reliable opinions on the perceptual quality are allowed. In addition, different annotator's behaviors in crowdsourcing are modeled, which allows us to identify the possibility that the annotator makes noisy labels during the test. The proposed model has been tested on both simulated data and realworld data, where it always shows superior performance than the other state-of-the-art models in terms of accuracy and robustness. CCS CONCEPTS • Information systems → Data mining; • Theory of computation → Data modeling.

show abstract

Semi-parametric analysis of multi-rater data

Cited by 12 publications

References 14 publications

The Benefits of a Model of Annotation

The Benefits of a Model of Annotation

Estimation of Discourse Segmentation Labels from Crowd Data

A Probabilistic Graphical Model for Analyzing the Subjective Visual Quality Assessment Data from Crowdsourcing

Contact Info

Product

Resources

About