Disentangling Item and Testing Effects in Inoculation Research on Online Misinformation: Solomon Revisited

Abstract

Online misinformation is a pervasive global problem. In response, psychologists have recently explored the theory of psychological inoculation: If people are preemptively exposed to a weakened version of a misinformation technique, they can build up cognitive resistance. This study addresses two unanswered methodological questions about a widely adopted online “fake news” inoculation game, Bad News. First, research in this area has often looked at pre- and post-intervention difference scores for the same items, which may imply that any observed effects are specific to the survey items themselves (item effects). Second, it is possible that using a pretest influences the outcome variable of interest, or that the pretest may interact with the intervention (testing effects). We investigate both item and testing effects in two online studies (total N = 2,159) using the Bad News game. For the item effect, we examine if inoculation effects are still observed when different items are used in the pre- and posttest. To examine the testing effect, we use a Solomon’s Three Group Design. We find that inoculation interventions are somewhat influenced by item effects, and not by testing effects. We show that inoculation interventions are effective at improving people’s ability to spot misinformation techniques and that the Bad News game does not make people more skeptical of real news. We discuss the larger relevance of these findings for evaluating real-world psychological interventions.

Breaking Harmony Square: A game that “inoculates” against political misinformation

“…In total, 8 of these posts were examples of "real" manipulative content found "in the wild" on social media and in fake news articles. The other 8 were social media posts that we created ("fictional fake news"), which were validated in previous research (Basol et al, 2020;Maertens et al, 2020;Roozenbeek, Maertens, et al, 2020). We did not hypothesize any significant differences between participants' assessments of "real" and "fictional" misinformation, but chose to include both types for the following reasons: 1) including "real" items increases the ecological validity of the study, as participants are tested on information that they could have encountered "in the wild"; 2) including "fictional" items maximizes experimental control and thus allows us to better isolate each manipulation technique and ensure political neutrality, and 3) by including "fictional" items, we account for the possibility that participants may have seen the "real" manipulative content before, a memory confound which could bias their assessment (Roozenbeek & van der Linden, 2019).…”

“…We chose Tetris because it is in the public domain, most people know how it works without practicing, and it involves about the same amount of cognitive effort as playing Harmony Square. Following the methodology established in prior research on "fake news" games (Basol et al, 2020;Maertens et al, 2020;Roozenbeek, Maertens, et al, 2020; Roozenbeek & van der Linden, 2019), we measured reliability judgments of social media posts containing misinformation, both before and after the intervention on a 1-7 Likert scale.…”

“…We therefore chose to focus on addressing the question of whether Harmony Square is effective at reducing susceptibility to political misinformation, rather than truth discernment (Pennycook et al, 2020) but we note that media literacy interventions can affect the rating of both credible and non-credible items (Guess et al, 2020). For a more detailed discussion on how gamified inoculation interventions affect people's perception of "real" (high-quality) news we refer the reader to Roozenbeek, Maertens et al (2020). Figure 5 shows an example of what our items look like in the survey environment.…”

Inoculating Against Fake News About COVID-19

“…A practical application of inoculation theory in the context of COVID-19 misinformation is the new online game, Go Viral!, 2 developed in collaboration with the United Kingdom government and the WHO in which players learn to resist three manipulation techniques commonly used to spread misinformation about the coronavirus: fearmongering, the use of fake experts, and conspiracy theories. An open question in active inoculation research is the extent to which inoculation can boost truth-discernment skills, that is, not just the ability to spot and resist misinformation attacks but also the ability to better identify real or credible news (Guess et al, 2020;Roozenbeek et al, 2020a). Compton et al (2016) called for more "work that pushes forward our understanding of persuasion and has applied value as a health messaging strategy to help combat serious threats to healthy living" (p. 1).…”

The Psychology of Fake News

“…One potentially promising alternative class of interventions involve a more proactive "inoculation" or "prebunking" against misinformation [75,114]. In one example, the "Bad News Game" uses a 10-20 minute interactive tutorial to teach people how to identify fake news in an engaging way [115,116]. An important limitation of such approaches is that they are "opt in"that is, people have to actively choose to engage with the inoculation technique (often for a fairly substantial amount of timeat least in terms of the internet attention span; [117]).…”