Gamified inoculation interventions do not improve discrimination between true and fake news: Reanalyzing existing research with receiver operating characteristic analysis.

Abstract

Gamified inoculation interventions designed to improve the detection of online misinformation are becoming increasingly prevalent. Two of the most notable interventions of this kind are Bad News and Go Viral!. To assess their efficacy, prior research has typically used pre–post designs in which participants rated the reliability or manipulativeness of true and fake news items before and after playing these games, while most of the time also including a control group who played an irrelevant game (Tetris) or did nothing at all. Mean ratings were then compared between pre-tests and post-tests and/or between the control and experimental conditions. Critically, these prior studies have not separated response bias effects (overall tendency to respond “true” or “fake”) from discrimination (ability to distinguish between true and fake news, commonly dubbed discernment). We reanalyzed the results from five prior studies using receiver operating characteristic (ROC) curves, a method common to signal detection theory that allows for discrimination to be measured free from response bias. Across the studies, when comparable true and fake news items were used, Bad News and Go Viral! did not improve discrimination, but rather elicited more “false” responses to all news items (more conservative responding). These novel findings suggest that the current gamified inoculation interventions designed to improve fake news detection are not as effective as previously thought and may even be counterproductive. They also demonstrate the usefulness of ROC analysis, a largely unexploited method in this setting, for assessing the effectiveness of any intervention designed to improve fake news detection.

Is the COVID-19 Fake News game good news? Testing whether creating and disseminating fake news about vaccines in a computer game reduces people's belief in anti-vaccine arguments

“…Results, however, showing the high effectiveness of games like “Bad News” or “GoViral!” have been questioned before ( Modirrousta-Galian & Higham, 2023 ). It has been pointed out (using signal detection theory) that people, after playing such a game, are more suspicious of any information, not only false but also true, unless the true information was obvious (e.g., something that may have been heard many times in mainstream media).…”

Perceiving as biased but nevertheless persuaded? Effects of fact–checking news delivered by partisan media

“…Last but not least, this diverging pattern may help further illuminate why individuals continue to hold inaccurate beliefs even when they acknowledge the initial misinformation is false questionable in its veracity. Recent corrections to misinformation often do not entirely the effect of misinformation in subsequent judgmental tasks (for meta-analytic summary, see Walter & Tukachinsky, 2020), has little to do with sharing and using such information onwards (Sirlin et al, 2021), or even makes individuals skeptical to any information regardless of its veracity (Modirrousta-Galian & Higham, 2023). In line with these troubling findings, our work also suggests that perceiving fact-checks as biased may undermine its potential influence in the long run, as it may invite further cognitive resistance and avoidance strategies such as counter-arguing and inattention despite momentary persuasive effect in the short run.…”

Exposure to Higher Rates of False News Erodes Media Trust and Fuels Overconfidence

“…For a detailed explanation of these methods and why they are suitable for misinformation research, see (Modirrousta-Galian & Higham, 2023).…”

“…What sort of cues might achieve this broad skepticism in the media environment? Rhetorical claims about "fake news" from elected officials (Van Duyn & Collier, 2019), warnings about the prevalence of misinformation (Clayton et al, 2020;Hoes et al, 2022;van der Meer et al, 2023), educational games to help people spot manipulation techniques (Modirrousta-Galian & Higham, 2023), exposure to highly implausible statements (Ulusoy et al, 2021) or misinformation (van der Meer et al 2023), for example, appear to shift mindsets in this way, and may have unintended effects on belief in credible information. A news environment consisting primarily of false news might result in the same effect.…”