Learning with Different Amounts of Annotation: From Zero to Many Labels

Abstract

Training NLP systems typically assumes access to annotated data that has a single human label per example. Given imperfect labeling from annotators and inherent ambiguity of language, we hypothesize that single label is not sufficient to learn the spectrum of language interpretation. We explore new annotation distribution schemes, assigning multiple labels per example for a small subset of training examples. Introducing such multi label examples at the cost of annotating fewer examples brings clear gains on natural language inference task and entity typing task, even when we simply first train with a single label data and then fine tune with multi label examples. Extending a MixUp data augmentation framework, we propose a learning algorithm that can learn from training examples with different amount of annotation (with zero, one, or multiple labels). This algorithm efficiently combines signals from uneven training data and brings additional gains in low annotation budget and cross domain settings. Together, our method achieves consistent gains in two tasks, suggesting distributing labels unevenly among training examples can be beneficial for many NLP tasks. 1

Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance

“…Another study by (Bai et al, 2021) framed domain adaptation with a constrained budget as a consumer choice problem and evaluated the utility of different combinations of pretraining and data annotation under varying budget constraints. Another study by (Zhang et al, 2021) explored new annotation distribution schemes, assigning multiple labels per example for a small subset of training examples, and proposed a learning algorithm that efficiently combines signals from uneven training data. Finally, a study by (Chen et al, 2022) proposed an approach that reserves a fraction of annotations to explicitly clean up highly probable error samples to optimize the annotation process.…”

“…This not only helps save annotation time and budget but also ensures efficient utilization of available resources. While some research (Wan et al, 2023;Zhang et al, 2021) has provided insights and suggestions on finding the optimal number of annotators, a definitive solution to this problem has yet to be achieved.…”

Natural Language Deduction through Search over Statement Compositions

“…Future work could expand the training data of our NLI model to account for the subjectivity of NLI judgments (Pavlick and Kwiatkowski, 2019;Nie et al, 2020), particularly by modifying our data collection procedure (Zhang et al, 2021).…”

Clean or Annotate: How to Spend a Limited Data Collection Budget

“…Bai et al (2021) show that similar trade-offs exist when performing domain adaptation on a constrained budget. Zhang et al (2021) observe that difficult examples benefit from additional annotations, so optimal spending actually varies the amount of labels given to each example. Our approach actively targets examples for relabeling based on its likelihood of noise, whereas they randomly select examples for multi-labeling without considering its characteristics.…”