All that glitters...

Abstract

Evaluation has emerged as a central concern in natural language processing (NLP) over the last few decades. Evaluation is done against a gold standard, a manually linguistically annotated dataset, which is assumed to provide the ground truth against which the accuracy of the NLP system can be assessed automatically. In this article, some methodological questions in connection with the creation of gold standard datasets are discussed, in particular (non-)expectations of linguistic expertise in annotators and the interannotator agreement measure standardly but unreflectedly used as a kind of quality index of NLP gold standards.

Annotation for computational argumentation analysis: Issues and perspectives

“…This is part of a larger set of issues increasingly discussed in data‐driven NLP with regard to how linguistic data are annotated, such as the role of expert knowledge versus native‐speaker intuition (and perhaps more importantly: who counts as an expert), how to deal with variation in the annotations, and how to take factors such as these into account when setting up annotation tasks, as well as for calculating both IAA and machine learning accuracy (e.g. Babarczy et al., 2006; Bayerl & Paul, 2011; Borin, 2022; Gillick & Liu, 2010; Plank, 2022; Plank et al., 2014; Uma et al., 2021).…”