The Effects of Crowd Worker Biases in Fact-Checking Tasks

Abstract

Due to the increasing amount of information shared online every day, the need for sound and reliable ways of distinguishing between trustworthy and non-trustworthy information is as present as ever. One technique for performing fact-checking at scale is to employ human intelligence in the form of crowd workers. Although earlier work has suggested that crowd workers can reliably identify misinformation, cognitive biases of crowd workers may reduce the quality of truthfulness judgments in this context. We performed a systematic exploratory analysis of publicly available crowdsourced data to identify a set of potential systematic biases that may occur when crowd workers perform fact-checking tasks. Following this exploratory study, we collected a novel data set of crowdsourced truthfulness judgments to validate our hypotheses. Our findings suggest that workers generally overestimate the truthfulness of statements and that different individual characteristics (i.e., their belief in science) and cognitive biases (i.e., the affect heuristic and overconfidence) can affect their annotations. Interestingly, we find that, depending on the general judgment tendencies of workers, their biases may sometimes lead to more accurate judgments.

NewsComp: Facilitating Diverse News Reading through Comparative Annotation

“…Our result indicates that crowd annotation in subjective tasks is to a little extent affected by users' backgrounds-in our case, their news expertise, aligning with prior works [22,33,64]. Therefore, we can train users using their news expertise as a targeting criterion.…”

“…Example Response Perspectives and Biases (22) They were taking different sides of the equation and putting forward different thought processes Information Placement or Depth (14) The right article was less descriptive and focused more on the restrictions and not the case. It was definitely telling the story from one point of view.…”

Groups of experts often differ in their decisions: What are the implications for AI and machine learning? A commentary onNoise: A Flaw in Human Judgment, byKahneman, Sibony, and Sunstein (2021)

“…Building on earlier work by Kahneman and colleagues, a number of studies have also addressed biases in crowdsourced judgments, for example, Eickhoff (2018) and Draws et al. (2022). Based on proposals by Kahneman, Lovallo, and Sibony (2011) for reducing biases in individuals and small groups Draws et al.…”

“…A number of studies have addressed psychological factors in crowdsourcing, such as effects of priming (Harrison et al 2013) and (Morris, Dontcheva, and Gerber 2012); task design (Alagarai, Rajeshuni, and Indurkhya 2014); task breaks (Dai et al 2015); task assignment based on individual difference measures (Hettiachchi et al 2020). Building on earlier work by Kahneman and colleagues, a number of studies have also addressed biases in crowdsourced judgments, for example, Eickhoff (2018) and Draws et al (2022). Based on proposals by Kahneman, Lovallo, and Sibony (2011) for reducing biases in individuals and small groups Draws et al (2021) developed a checklist for reducing cognitive biases in crowdsourcing.…”

Modeling and Mitigating Online Misinformation: a Suggested Blockchain Approach

“…These organizations are often centralized, meaning that they are controlled by a small group of people. This can make it difficult for them to guarantee transparency and prevent manipulation, such as cherry-picking in favor of a particular party [5,15,54]. In addition, "fact-checking" is labor-intensive, and the curators may not be able to keep up with emerging news stories or cover the entire news ecosystem.…”