Clean or Annotate: How to Spend a Limited Data Collection Budget

Abstract

Crowdsourcing platforms are often used to collect datasets for training machine learning models, despite higher levels of inaccurate labeling compared to expert labeling. There are two common strategies to manage the impact of such noise: The first involves aggregating redundant annotations, but comes at the expense of labeling substantially fewer examples. Secondly, prior works have also considered using the entire annotation budget to label as many examples as possible and subsequently apply denoising algorithms to implicitly clean the dataset. We find a middle ground and propose an approach which reserves a fraction of annotations to explicitly clean up highly probable error samples to optimize the annotation process. In particular, we allocate a large portion of the labeling budget to form an initial dataset used to train a model. This model is then used to identify specific examples that appear most likely to be incorrect, which we spend the remaining budget to relabel. Experiments across three model variations and four natural language processing tasks show our approach outperforms or matches both label aggregation and advanced denoising methods designed to handle noisy labels when allocated the same finite annotation budget.

EmoAtlas: An emotional profiling tool merging psychological lexicons, artificial intelligence and network science

“…Tables 3 and 4 report accuracy, precision and recall scores for the 6 classification tasks relative to detecting the presence or absence of a given emotion in a post from GoEmotions 34 . Recent work 71 showed that the original labelling from GoEmotions, over 27 emotional states, contained several invalid annotations. Since there is no evidence that the errors are systematically biased to favour one method over the others, we preserved the original labelling but considered only those labels equivalent to our emotions rather than mapping all 27 emotional states into the emotions supported by BERT (as done in 34 ) or EmoAtlas.…”

EmoAtlas: An emotional network analyser of texts merging psychological lexicons, artificial intelligence and network science

“…Tables 4, 5 and 6 report accuracy, precision and recall scores for the 6 classification tasks relative to detecting the presence or absence of a given emotion in a post from GoEmotions [18]. Recent work [12] showed that the original labelling from GoEmotions, over 27 emotional states, contained several invalid annotations. Since there is no evidence that the errors are systematically biased to favour one method over the others, we preserved the original labelling but considered only those labels equivalent to our emotions rather than mapping all 27 emotional states into the emotions supported by BERT, as done in [18], or EmoAtlas.…”

Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance

“…Another study by (Zhang et al, 2021) explored new annotation distribution schemes, assigning multiple labels per example for a small subset of training examples, and proposed a learning algorithm that efficiently combines signals from uneven training data. Finally, a study by (Chen et al, 2022) proposed an approach that reserves a fraction of annotations to explicitly clean up highly probable error samples to optimize the annotation process. All these studies contribute to the understanding of how to maximize the performance of NLP models under restricted annotation budgets.…”