Learning from crowds with variational Gaussian processes

Ruiz, Pablo; Morales-Álvarez, Pablo; Molina, Rafael; Katsaggelos, Aggelos K.

doi:10.1016/j.patcog.2018.11.021

Cited by 54 publications

(18 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MTL correct incorrect STL correct 6.614 5.961 incorrect 6.015* 5.727 Dawid and Skene (1979), who proposed an Expectation-Maximization (EM) based aggregation model. This model has since influenced a large body of work on annotation aggregation, and modeling annotator competence (Carpenter et al, 2009;Hovy et al, 2013;Raykar et al, 2010;Paun et al, 2018;Ruiz et al, 2019). In our experiments on POS-tagging, we evaluated the possibility of testing Dawid-Skene labels rather than Majority Voting, finding that the performance of the two against the gold standard was mostly the same.…”

Section: Error Analysismentioning

confidence: 99%

“…In our experiments on POS-tagging, we evaluated the possibility of testing Dawid-Skene labels rather than Majority Voting, finding that the performance of the two against the gold standard was mostly the same. Some of these methods also evaluate the annotators' expertise (Dawid and Skene, 1979;Raykar et al, 2010;Hovy et al, 2013;Ruiz et al, 2019). Others just penalize disagreement (Pan et al, 2019).…”

Section: Error Analysismentioning

confidence: 99%

“…Usually, the labels used in NLP classification tasks are produced by sets of human annotators. As disagreement between annotators is common, many methods aggregate the different answers into a supposedly correct one (Dawid and Skene, 1979;Carpenter, 2008;Hovy et al, 2013;Raykar et al, 2010;Paun et al, 2018;Ruiz et al, 2019). However, the aggregated labels obtained in this way mask the world's real complexity: instances can be intrinsically ambiguous (Poesio and Artstein, 2005;Zeman, 2010;Pavlick and Kwiatkowski, 2019), or so challenging to evaluate that considerable disagreement between different annotators is unavoidable.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning

Fornaciari¹,

Uma²,

Paun³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Supervised learning assumes that a ground truth label exists. However, the reliability of this ground truth depends on human annotators, who often disagree. Prior work has shown that this disagreement can be helpful in training models. We propose a novel method to incorporate this disagreement as information: in addition to the standard error computation, we use soft labels (i.e., probability distributions over the annotator labels) as an auxiliary task in a multi-task neural network. We measure the divergence between the predictions and the target soft labels with several loss-functions and evaluate the models on various NLP tasks. We find that the soft-label prediction auxiliary task reduces the penalty for errors on ambiguous entities and thereby mitigates overfitting. It significantly improves performance across tasks beyond the standard approach and prior work.

show abstract

Section: Error Analysismentioning

confidence: 99%

Section: Error Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning

Fornaciari¹,

Uma²,

Paun³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…Several GP-based models have been proposed for label aggregation task [59,120,134,137]. Groot et al [59] studied the problem of inferring the true real-valued label given multiple noisy real-valued labels.…”

Section: Gaussian Process For Aggregating Crowd Annotationsmentioning

confidence: 99%

Effective collection construction for information retrieval evaluation and optimization

2020

SIGIR Forum

View full text Add to dashboard Cite

The availability of test collections in Cranfield paradigm has significantly benefited the development of models, methods and tools in information retrieval. Such test collections typically consist of a set of topics, a document collection and a set of relevance assessments. Constructing these test collections requires effort of various perspectives such as topic selection, document selection, relevance assessment, and relevance label aggregation etc. The work in the thesis provides a fundamental way of constructing and utilizing test collections in information retrieval in an effective, efficient and reliable manner. To that end, we have focused on four aspects. We first study the document selection issue when building test collections. We devise an active sampling method for efficient large-scale evaluation [Li and Kanoulas, 2017]. Different from past sampling-based approaches, we account for the fact that some systems are of higher quality than others, and we design the sampling distribution to over-sample documents from these systems. At the same time, the estimated evaluation measures are unbiased, and assessments can be used to evaluate new, novel systems without introducing any systematic error. Then a natural further step is determining when to stop the document selection and assessment procedure. This is an important but understudied problem in the construction of test collections. We consider both the gain of identifying relevant documents and the cost of assessing documents as the optimization goals. We handle the problem under the continuous active learning framework by jointly training a ranking model to rank documents, and estimating the total number of relevant documents in the collection using a "greedy" sampling method [Li and Kanoulas, 2020]. The next stage of constructing a test collection is assessing relevance. We study how to denoise relevance assessments by aggregating from multiple crowd annotation sources to obtain high-quality relevance assessments. This helps to boost the quality of relevance assessments acquired in a crowdsourcing manner. We assume a Gaussian process prior on query-document pairs to model their correlation. The proposed model shows good performance in terms of interring true relevance labels. Besides, it allows predicting relevance labels for new tasks that has no crowd annotations, which is a new functionality of CrowdGP. Ablation studies demonstrate that the effectiveness is attributed to the modelling of task correlation based on the axillary information of tasks and the prior relevance information of documents to queries. After a test collection is constructed, it can be used to either evaluate retrieval systems or train a ranking model. We propose to use it to optimize the configuration of retrieval systems. We use Bayesian optimization approach to model the effect of a δ -step in the configuration space to the effectiveness of the retrieval system, by suggesting to use different similarity functions (covariance functions) for continuous and categorical values, and examine their ability to effectively and efficiently guide the search in the configuration space [Li and Kanoulas, 2018]. Beyond the algorithmic and empirical contributions, work done as part of this thesis also contributed to the research community as the CLEF Technology Assisted Reviews in Empirical Medicine Tracks in 2017, 2018, and 2019 [Kanoulas et al., 2017, 2018, 2019]. Awarded by: University of Amsterdam, Amsterdam, The Netherlands. Supervised by: Evangelos Kanoulas. Available at: https://dare.uva.nl/search?identifier=3438a2b6-9271-4f2c-add5-3c811cc48d42.

show abstract

“…As EM solves a non-convex problem that is susceptible to initialization, moment-based methods [7][8][9][10] have been advocated to Work in this paper was supported by NSF grant 1901134. Emails: traga003@umn.edu, georgios@umn.edu initialize it; see also [11][12][13] for alternatives that account for dependent data. Regarding adversarial attacks in unsupervised ensembles and crowdsourcing, [14] modified the EM algorithm of [6] to detect and eliminate spammers during the aggregation phase.…”

Section: Introductionmentioning

confidence: 99%

Identifying Spammers to Boost Crowdsourced Classification

Traganitis

Giannakis

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

The present work addresses the problem of adversarial attacks in unsupervised ensemble or crowdsourcing classification tasks. Under certain conditions, it is shown, both analytically and through numerical tests, that spammers cause the most damage with respect to classification performance. To curb their effect, a novel spectral algorithm for spammer detection that utilizes second-order statistics of annotators, is developed and preliminary results on synthetic and real data showcase the potential of this approach.

show abstract

Learning from crowds with variational Gaussian processes

Cited by 54 publications

References 18 publications

Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning

Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning

Effective collection construction for information retrieval evaluation and optimization

Identifying Spammers to Boost Crowdsourced Classification

Contact Info

Product

Resources

About