Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement

Sommerauer, Pia; Fokkens, Antske; Vossen, Piek

doi:10.18653/v1/2020.coling-main.422

Cited by 5 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given identical instructions and identical items, some annotators may focus on different attributes of the item or have a different interpretation of the labeling criteria. Understanding and modelling label uncertainty remains a compelling challenge in evaluating machine learning systems (Sommerauer, Fokkens, and Vossen, 2020;Resnick et al, 2021).…”

Section: Developments Inmentioning

confidence: 99%

Proceedings of the First Workshop on Dynamic Adversarial Data Collection

2022

View full text Add to dashboard Cite

This volume contains papers from the First Workshop on Dynamic Adversarial Data Collection (DADC), held at NAACL 2022. Dynamic Adversarial Data Collection (DADC) has been gaining traction in the community as a promising approach to improving data collection practices, model evaluation and performance. DADC allows us to collect human-written data dynamically with models in the loop. Humans can be tasked with finding adversarial examples that fool current state-of-the-art models (SOTA), for example, or they can cooperate with models to find interesting examples. This offers two benefits: it allows us to gauge how good contemporary SOTA methods really are; and it yields data that may be used to train even stronger models by specifically targeting their current weaknesses. The first workshop on DADC and corresponding shared task focus on three currently under-explored themes: i) understanding how humans can be incentivized to creatively identify and target model weaknesses to increase their chances of fooling the model; ii) how humans and machines can cooperate to produce the most useful data; and iii) how the interaction between humans and machines can further drive performance improvements, both from the perspectives of traditional evaluation metrics as well as those of robustness and fairness. iv

show abstract

Section: Developments Inmentioning

confidence: 99%

Proceedings of the First Workshop on Dynamic Adversarial Data Collection

2022

View full text Add to dashboard Cite

show abstract

“…Recent studies which investigated disagreements in natural language inference (Pavlick and Kwiatkowski, 2019) and semantic annotation (Sommerauer et al, 2020) claim that disagreement in natural language evaluation is often expected due to ambiguity and variation of language. Therefore, a number of disagreements do not represent "errors" or "noise" but are fully legitimate.…”

Section: Related Workmentioning

confidence: 99%

Agree to Disagree: Analysis of Inter-Annotator Disagreements in Human Evaluation of Machine Translation Output

Popović¹

2021

Proceedings of the 25th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

This work describes an analysis of interannotator disagreements in human evaluation of machine translation output. The errors in the analysed texts were marked by multiple annotators under guidance of different quality criteria: adequacy, comprehension, and an unspecified generic mixture of adequacy and fluency. Our results show that different criteria result in different disagreements, and indicate that a clear definition of quality criterion can improve the inter-annotator agreement. Furthermore, our results show that for certain linguistic phenomena which are not limited to one or two words (such as word ambiguity or gender) but span over several words or even entire phrases (such as negation or relative clause), disagreements do not necessarily represent "errors" or "noise" but are rather inherent to the evaluation process. On the other hand, for some other phenomena (such as omission or verb forms) agreement can be easily improved by providing more precise and detailed instructions to the evaluators.

show abstract

Section: Introductionmentioning

confidence: 99%

Posthoc Verification and the Fallibility of the Ground Truth

Ding¹,

Botzer²,

Weninger³

2021

Preprint

View full text Add to dashboard Cite

Classifiers commonly make use of pre-annotated datasets, wherein a model is evaluated by pre-defined metrics on a held-out test set typically made of human-annotated labels. Metrics used in these evaluations are tied to the availability of well-defined ground truth labels, and these metrics typically do not allow for inexact matches. These noisy ground truth labels and strict evaluation metrics may compromise the validity and realism of evaluation results. In the present work, we discuss these concerns and conduct a systematic posthoc verification experiment on the entity linking (EL) task. Unlike traditional methodologies, which asks annotators to provide free-form annotations, we ask annotators to verify the correctness of annotations after the fact (i.e., posthoc). Compared to pre-annotation evaluation, state-of-the-art EL models performed extremely well according to the posthoc evaluation methodology. Posthoc validation also permits the validation of the ground truth dataset. Surprisingly, we find predictions from EL models had a similar or higher verification rate than the ground truth. We conclude with a discussion on these findings and recommendations for future evaluations.

show abstract

Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement

Cited by 5 publications

References 20 publications

Proceedings of the First Workshop on Dynamic Adversarial Data Collection

Proceedings of the First Workshop on Dynamic Adversarial Data Collection

Agree to Disagree: Analysis of Inter-Annotator Disagreements in Human Evaluation of Machine Translation Output

Posthoc Verification and the Fallibility of the Ground Truth

Contact Info

Product

Resources

About