Annotating Social Media Data From Vulnerable Populations: Evaluating Disagreement Between Domain Experts and Graduate Student Annotators

Abstract

Researchers in computer science have spent considerable time developing methods to increase the accuracy and richness of annotations. However, there is a dearth in research that examines the positionality of the annotator, how they are trained and what we can learn from disagreements between different groups of annotators. In this study, we use qualitative analysis, statistical and computational methods to compare annotations between Chicago-based domain experts and graduate students who annotated a total of 1,851 tweets with images that are a part of a larger corpora associated with the Chicago Gang Intervention Study, which aims to develop a computational system that detects aggression and loss among gang-involved youth in Chicago. We found evidence to support the study of disagreement between annotators and underscore the need for domain expertise when reviewing Twitter data from vulnerable populations. Implications for annotation and content moderation are discussed.

Toward Gender-Inclusive Coreference Resolution

“…Both datasets show significant gaps in system performance, but perhaps moreso, show that taking crowdworker judgments as "gold standard" can be problematic. It may be the case that to truly build gender inclusive datasets and systems, we need to hire or consult experiential experts (Patton et al, 2019;Young et al, 2019).…”

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries

“…– The interpretation and assessment of results are too often done by data experts, not by domain experts . This is problematic as there are known differences in how non-experts and experts interact with and validate systems outputs (White et al, 2009 ; Patton et al, 2019 ), particularly for critical application domains such as health. Furthermore, to interpret, e.g., the relations found by causal inference techniques as causal, among others, unobserved covariates are assumed ignorable; yet, without significant domain expertise this cannot be asserted (see section 7.4).…”

“…Though the goal of an assessment task is to provide human input, underspecification or appeal to subjective judgment can introduce unintended biases that are often hard to detect. In fact, for many annotations tasks, the characteristics of those that do the annotations can significantly influence how they annotate (Olteanu et al, 2017a ; Patton et al, 2019 ).…”

Duluth at SemEval-2020 Task 12: Offensive Tweet Identification in English with Logistic Regression

“…In general it would appear virtually impossible to reliably annotate data without some background knowledge about the participants in the dialogue, as well as larger cultural contexts that might be at work (e.g, (Patton et al, 2019;Frey et al, 2020)). That said we believe that annotated corpora is an important resource for this problem, and we need to continue to refine our processes for the creation of the same.…”