A Neural Model for Aggregating Coreference Annotation in Crowdsourcing

Abstract

Coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity. Collecting sufficient labelled data from expert annotators to train a highperformance coreference resolution system is time-consuming and expensive. Crowdsourcing makes it possible to obtain the required amounts of data rapidly and cost-effectively. However, crowd-sourced labels can be noisy. To ensure high-quality data, it is crucial to infer the correct labels by aggregating the noisy labels. In this paper, we split the aggregation into two subtasks, i.e, mention classification and coreference chain inference. Firstly, we predict the general class of each mention using an autoencoder, which incorporates contextual information about each mention, while at the same time taking into account the mention's annotation complexity and annotators' reliability at different levels. Secondly, to determine the coreference chain of each mention, we use weighted voting which takes into account the learned reliability in the first subtask. Experimental results demonstrate the effectiveness of our method in predicting the correct labels. We also illustrate our model's interpretability through a comprehensive analysis of experimental results.

Identifying Chinese Opinion Expressions with Extremely-Noisy Crowdsourcing Annotations

“…A number of models are developed to aggregate a higher-quality corpus from the crowdsourcing corpus (Raykar et al, 2010;Rodrigues et al, 2014a,b;Moreno et al, 2015), aiming to reduce the gap over the expert-annotated corpus. Recently, modeling the bias between the crowd annotators and the oracle experts has been demonstrated effectively (Nguyen et al, 2017;Simpson and Gurevych, 2019;Li et al, 2020), focusing on the label bias between the crowdsourcing annotations and gold-standard answers, regarding crowd-sourcing annotations as annotator-sensitive noises. do not hold crowdsourcing annotations as noisy labels, while regard them as ground-truths by the understanding of individual crowd annotators.…”

Paladin: an annotation tool based on active and proactive learning

“…But in reality, different annotators have levels of expertise in different domains. It has been demonstrated that proactive learning is helpful for task allocation in crowdsourcing setting where the level of expertise varies from annotator to annotator (Donmez and Carbonell, 2010;Li et al, 2017Li et al, , 2019Li et al, , 2020b. Proactive learning is useful in modelling the annotator reliability which can used to assign the unlabelled instances to the best possible Before any annotation, Paladin estimates the proficiency of the annotators for each class by assigning the documents in the seed dataset to all annotators.…”

Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning

“…Research in NLP has noticed the inherent disagreement and ambiguity in annotations since long ago (Poesio and Artstein, 2005;Zeman, 2010). Many previous works expose and learn from ambiguous, noisy annotations in coreference resolution (Poesio et al, 2008(Poesio et al, , 2019Li et al, 2020). Related research also exists in POS-tagging (Zeman, 2010;Plank et al, 2014Plank et al, , 2016, semantic frame disambiguation (Dumitrache et al, 2019), humorousness prediction (Simpson et al, 2019), etc.…”