Supervised learning from multiple experts

Raykar, Vikas C.; Yu, Shipeng; Zhao, Linda; Jerebko, Anna K.; Florin, Charles; Valadez, Gerardo Hermosillo; Bogoni, Luca; Moy, Linda

doi:10.1145/1553374.1553488

Cited by 272 publications

(131 citation statements)

References 10 publications

Supporting

Mentioning

131

Contrasting

Order By: Relevance

“…In addition, BCLAs was compared to BCLA, 'smart fusion' (i.e. the benchmarking algorithm proposed by authors of the dataset (Karlen et al 2013)), the bestperforming (lowest-MAE) single algorithm (denoted 'Best'), the EM algorithm proposed by Raykar et al (2010) (denoted as EM-R), the scalar Simultaneous Truth and Performance Level Estimation (denoted as sSTAPLE) proposed by Warfield et al (2008), and also the mean and median voting approaches.…”

Section: Capnobase Rr Datasetmentioning

confidence: 99%

Bayesian fusion of physiological measurements using a signal quality extension

Zhu

Johnson²,

Yang³

et al. 2018

Physiol. Meas.

View full text Add to dashboard Cite

Objective: The fusion of multiple noisy labels for biomedical data (such as ECG annotations, which may be obtained from human experts or from automated systems) into a single robust annotation has many applications in physiologic monitoring. Directly modelling the difficulty of the task has the potential to improve the fusion of such labels. This paper proposes a means for the incorporation of task difficulty, as quantified by ‘signal quality’, into the fusion process. Approach: We propose a Bayesian fusion model to infer a consensus through aggregating labels, where the labels are provided by multiple imperfect automated algorithms (or ‘annotators’). Our model incorporates the signal quality of the underlying recording when fusing labels. We compare our proposed model with previously published approaches. Two publicly available datasets were used to demonstrate the feasibility of our proposed model: one focused on QT interval estimation in the ECG and the other focused on respiratory rate (RR) estimation from the photoplethysmogram (PPG). We inferred the hyperparameters of our model using maximum- a posteriori inference and Gibbs sampling. Main results: For the QT dataset, our model significantly outperformed the previously published models (root-mean-square error of ms for our model versus ms from the best existing model) when fusing labels from only three annotators. For the RR dataset, no improvement was observed compared to the same model without signal quality modelling, where our model outperformed existing models (mean-absolute error of bpm for our model versus bpm from the best existing model). We conclude that our approach demonstrates the feasibility of using a signal quality metric as a confidence measure to improve label fusion. Significance: Our Bayesian learning model provides an extension over existing work to incorporate signal quality as a confidence measure to improve the reliability of fusing labels from biomedical datasets.

show abstract

Section: Capnobase Rr Datasetmentioning

confidence: 99%

Bayesian fusion of physiological measurements using a signal quality extension

Zhu

Johnson²,

Yang³

et al. 2018

Physiol. Meas.

View full text Add to dashboard Cite

show abstract

“…The growth of crowdwork applications has catalyzed interest in two related but distinct research questions concerning human evaluations: inferring true labels from multiple annotations [1]- [7] and estimating evaluator expertise [8]- [12].…”

Section: Related Workmentioning

confidence: 99%

“…There have been attempts at unifying the two research directions by modeling evaluators and items [9,15,16,17]. [9] associates an expertise level with individual evaluators and a difficulty level with each of the items; the probability that an evaluator rates an item correctly is then modeled as a function of these two parameters.…”

Section: Related Workmentioning

confidence: 99%

“…Much of the earlier work relies on stronger assumptions about the data. For instance, [17] and [8] assume that each item has been annotated by most of the evaluators under consideration, which may not be the case in many natural settings. Approaches that work with evaluator-item pairs (e.g.…”

Section: Related Workmentioning

confidence: 99%

“…A long line of research has studied how to take multiple labels from non-expert evaluators and synthesize them into a single high-quality label [1]- [7], and how to estimate the performance of individual evaluators on various tasks [8]- [12]. However, relatively little attention has been focused on obtaining deeper insights into evaluations such as understanding the characteristics of mistakes being made and identifying the shared attributes of evaluators and items that are relevant to the quality of the resulting label.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Bayesian Framework for Modeling Human Evaluations

Lakkaraju

Leskovec

Kleinberg

et al. 2015

Proceedings of the 2015 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

Several situations that we come across in our daily lives involve some form of evaluation: a process where an evaluator chooses a correct label for a given item. Examples of such situations include a crowd-worker labeling an image or a student answering a multiple-choice question. Gaining insights into human evaluations is important for determining the quality of individual evaluators as well as identifying true labels of items. Here, we generalize the question of estimating the quality of individual evaluators, extending it to obtain diagnostic insights into how various evaluators label different kinds of items. We propose a series of increasingly powerful hierarchical Bayesian models which infer latent groups of evaluators and items with the goal of obtaining insights into the underlying evaluation process. We apply our framework to a wide range of real-world domains, and demonstrate that our approach can accurately predict evaluator decisions, diagnose types of mistakes evaluators tend to make, and infer true labels of items.

show abstract