Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks

Röttger, Paul; Vidgen, Bertie; Hovy, Dirk; Pierrehumbert, Janet B.

doi:10.18653/v1/2022.naacl-main.13

Cited by 42 publications

(35 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, researchers commonly use some notion of the agreement to measure the task's subjectiveness, such as using inter-annotator agreement metrics Cohen's Kappa (Cohen 1960) or Fleiss' Kappa (Fleiss 1971) to measure annotations' reliability. But when presenting the final results in the downstream task, people usually use the aggregated labels that can conceal informative disagreement and evaluation metrics that are unaware of the task's subjectiveness (Röttger et al 2022).…”

Section: Related Workmentioning

confidence: 99%

Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information

Wan¹,

Kim²,

Kang³

2023

Preprint

View full text Add to dashboard Cite

In NLP annotation, it is common to have multiple annotators label the text and then obtain the ground truth labels based on the agreement of major annotators. However, annotators are individuals with different backgrounds, and minors' opinions should not be simply ignored. As annotation tasks become subjective and topics are controversial in modern NLP tasks, we need NLP systems that can represent people's diverse voices on subjective matters and predict the level of diversity. This paper examines whether the text of the task and annotators' demographic background information can be used to estimate the level of disagreement among annotators. Particularly, we extract disagreement labels from the annotators' voting histories in the five subjective datasets, and then fine-tune language models to predict annotators' disagreement. Our results show that knowing annotators' demographic information, like gender, ethnicity, and education level, helps predict disagreements. In order to distinguish the disagreement from the inherent controversy from text content and the disagreement in the annotators' different perspectives, we simulate everyone's voices with different combinations of annotators' artificial demographics and examine its variance of the finetuned disagreement predictor. Our paper aims to improve the annotation process for more efficient and inclusive NLP systems through a novel disagreement prediction mechanism. Our code and dataset are publicly available. 1 * This work was done while RW and JK were at the Minnesota NLP lab.

show abstract

Section: Related Workmentioning

confidence: 99%

Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information

Wan¹,

Kim²,

Kang³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…While in some cases it may help to resolve differences between annotators (Hagerer et al, 2021), it is often insightful to acknowledge and explore the subjectivity of labels assigned by people or groups (Leonardelli et al, 2021;Sap et al, 2022). A dataset with labels from individuals, termed descriptive annotations, will help us build models to better understand differences in people's views of socially acceptable behavior (Röttger et al, 2022). Lourie et al (2021) first examined AITA, suggesting that the descriptive ethics contained in people's judgements could serve as a valuable resource for developing machines that can appropriately and safely interact with people.…”

Section: Related Workmentioning

confidence: 99%

Understanding Interpersonal Conflict Types and their Impact on Perception Classification

Welch¹,

Plepi²,

Neuendorf³

et al. 2022

Preprint

View full text Add to dashboard Cite

Studies on interpersonal conflict have a long history and contain many suggestions for conflict typology. We use this as the basis of a novel annotation scheme and release a new dataset of situations and conflict aspect annotations. We then build a classifier to predict whether someone will perceive the actions of one individual as right or wrong in a given situation, outperforming previous work on this task. Our analyses include conflict aspects, but also generated clusters, which are human validated, and show differences in conflict content based on the relationship of participants to the author. Our findings have important implications for understanding conflict and social norms.

show abstract

“…Recent research suggests embracing disagreements by developing multi-annotator architectures that capture differences in annotator perspective (Davani et al, 2022;Basile et al, 2021;Uma et al, 2021). While this approach better models how abuse is perceived, it is not suitable for content moderation where one has to decide whether to remove a post and a prescriptive paradigm is preferable (Rottger et al, 2022). Zufall et al (2020) adopt a more objective approach, as they aim to detect content that is illegal according to EU legislation.…”

Section: Related Workmentioning

confidence: 99%

Explainable Abuse Detection as Intent Classification and Slot Filling

Calabrese¹,

Roß²,

Lapata³

2022

Preprint

View full text Add to dashboard Cite

To proactively offer social media users a safe online experience, there is a need for systems that can detect harmful posts and promptly alert platform moderators. In order to guarantee the enforcement of a consistent policy, moderators are provided with detailed guidelines. In contrast, most stateof-the-art models learn what abuse is from labelled examples and as a result base their predictions on spurious cues, such as the presence of group identifiers, which can be unreliable. In this work we introduce the concept of policy-aware abuse detection, abandoning the unrealistic expectation that systems can reliably learn which phenomena constitute abuse from inspecting the data alone. We propose a machine-friendly representation of the policy that moderators wish to enforce, by breaking it down into a collection of intents and slots. We collect and annotate a dataset of 3,535 English posts with such slots, and show how architectures for intent classification and slot filling can be used for abuse detection, while providing a rationale for model decisions. 1

show abstract

Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks

Cited by 42 publications

References 20 publications

Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information

Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information

Understanding Interpersonal Conflict Types and their Impact on Perception Classification

Explainable Abuse Detection as Intent Classification and Slot Filling

Contact Info

Product

Resources

About