How Accurate Are Accuracy-Nudge Interventions? A Preregistered Direct Replication of Pennycook et al. (2020)

Abstract

As part of the Systematizing Confidence in Open Research and Evidence (SCORE) program, the present study consisted of a two-stage replication test of a central finding by Pennycook et al. (2020), namely that asking people to think about the accuracy of a single headline improves “truth discernment” of intentions to share news headlines about COVID-19. The first stage of the replication test ( n = 701) was unsuccessful ( p = .67). After collecting a second round of data (additional n = 882, pooled N = 1,583), we found a small but significant interaction between treatment condition and truth discernment (uncorrected p = .017; treatment: d = 0.14, control: d = 0.10). As in the target study, perceived headline accuracy correlated with treatment impact, so that treatment-group participants were less willing to share headlines that were perceived as less accurate. We discuss potential explanations for these findings and an unreported change in the hypothesis (but not the analysis plan) from the preregistration in the original study.

Digital literacy is associated with more discerning accuracy judgments but not sharing intentions

“…Although the hypothetical sharing intentions measure employed here has been widely used in the literature (e.g. Epstein et al 2021;Guess et al, 2020;Pennycook & Rand, 2019;Pennycook et al, 2020;Roozenbeek et al, 2021;Roozenbeek & van der Linden, 2020;Rosenzweig et al, 2021;Ross et al, 2021), it is of paramount importance for future work to examine the relationship between digital literacy and actual on-platform sharing.…”

Viruses, vaccines, and COVID-19: Explaining and improving risky decision-making.

“…Notably, this work has focused on “cognitive reflection,” which taps metacognitive monitoring among other cognitive abilities, suggesting that individuals simply pay little attention to the truth value of online content ( Pennycook & Rand, 2019 ). Consequently, when people are privately encouraged to consider the accuracy of the content they share, they are less likely to share false content (but also less likely to share true implausible content; Roozenbeek et al, 2021 ) although sharing is far from eliminated ( Pennycook et al, 2020 , Pennycook et al, 2021 ). These findings are typically explained in terms of standard dual-process theories discussed earlier, where “slow” effortful deliberative thinking corrects reasoning biases and fallacies produced by default processing.…”

Knowing Well, Being Well: well-being born of understanding: The Urgent Need for Coordinated and Comprehensive Efforts to Combat Misinformation

“…38,39 Only after researchers collected a sample of about twice the size of the original experiments did they find a small effect, at about 50% of the original study's effect size. 39 Second, accuracy primes can be moderated by partisanship such that they appear to be ineffective for people who strongly identify as conservative, at least in the United States. [38][39][40] Third, there is some indication that the effect conferred by accuracy primes occurs predominantly in the first few seconds after exposure (the treatment effect dissipated after participants rated about 7 headlines), although this initial finding requires further investigation.…”