It's All Relative! A Method to Counter Human Bias in Crowdsourced Stance Detection of News Articles

Abstract

Using human intelligence to identify news articles' political stances is common in research and practical applications. But human judgement can be biased and prone to errors stemming from the comprehension of tasks and political alignment. This paper proposes a relative rating method based on news articles' stances relative to raters' own stances to avoid comprehension inconsistency and to control for human bias in crowdsourced stance detection of news articles. We also show how to use the relative ratings to construct a measure for raters' stances on a political topic and to identify raters whose ratings are of higher quality than others. We implement our proposed methods in an online experiment that recruits Amazon Mechanical Turk users as raters for news articles on Gun Control. Using the data from the experiment, we find evidence that raters' own stances on Gun Control significantly impact ratings of related news articles, both at the individual levels and at the aggregate levels. We also present evidence that our relative-rating-based stance measure captures more information about raters' actual stances than their self-reported stance does.

APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPT

“…Data annotation commonly relies on human intelligence and suffers from challenges related to annotator quality, data comprehension, domain knowledge, and time cost [9,20,25,53]. Recent research has highlighted that prompt-based LLM applications, like ChatGPT, have the potential to substitute human intelligence as a data annotator for social computing tasks [64], e.g., sentiment analysis [4,64] and hate speech detection [26].…”