The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment

Abstract

Interest in the psychology of misinformation has exploded in recent years. Despite ample research, to date there is no validated framework to measure misinformation susceptibility. Therefore, we introduce Verificationdone, a nuanced interpretation schema and assessment tool that simultaneously considers Veracity discernment, and its distinct, measurable abilities (real/fake news detection), and biases (distrust/naïvité—negative/positive judgment bias). We then conduct three studies with seven independent samples (Ntotal = 8504) to show how to develop, validate, and apply the Misinformation Susceptibility Test (MIST). In Study 1 (N = 409) we use a neural network language model to generate items, and use three psychometric methods—factor analysis, item response theory, and exploratory graph analysis—to create the MIST-20 (20 items; completion time < 2 minutes), the MIST-16 (16 items; < 2 minutes), and the MIST-8 (8 items; < 1 minute). In Study 2 (N = 7674) we confirm the internal and predictive validity of the MIST in five national quota samples (US, UK), across 2 years, from three different sampling platforms—Respondi, CloudResearch, and Prolific. We also explore the MIST’s nomological net and generate age-, region-, and country-specific norm tables. In Study 3 (N = 421) we demonstrate how the MIST—in conjunction with Verificationdone—can provide novel insights on existing psychological interventions, thereby advancing theory development. Finally, we outline the versatile implementations of the MIST as a screening tool, covariate, and intervention evaluation framework. As all methods are transparently reported and detailed, this work will allow other researchers to create similar scales or adapt them for any population of interest.

Misleading But Not Fake: Measuring the Difference Between Manipulativeness Discernment and Veracity Discernment Using Psychometrically Validated Tests

“…In addition, participants had to compete the DEPICT Reliability Discernment Task (Reliability, 1-7; Maertens, Roozenbeek, et al, 2021) and the Misinformation Susceptibility Test (MIST, Fake/Real; Maertens, Götz, et al, 2023). Finally they were asked a set of demographics questions (education, political ideology, and party affiliation; age and gender was assessed by the panel provider at the beginning of the study), and received a debriefing.…”

“…As our focus for this work is on exploring manipulativeness discernment in contrast to veracity discernment (which is the main focus of Study 3), rather than scale development, we will not include in-depth discussions of the latest psychometric debates. As our approach mirrors the development protocol used by Maertens, Götz, et al (2023), we recommend their paper for a more in-depth discussion about the latest methods in psychometrics (and an in-depth discussion of the development of the MIST, which we followed to develop the MOCRI used in the present study).…”

“…But in the absence of psychometric validation, these item sets may not be representative of false (or misleading) information in a more general sense, which makes it difficult to compare results across studies, and-importantly for the present studyassess whether there are differences between why people believe and share explicit and implied misinformation. To address this question in the context of "fake news", Maertens, Götz, et al (2023) developed the 20-item Misinformation Susceptibility Test (MIST; see below), a psychometrically validated measure of people's ability to distinguish true from false news headlines. However, to date, no similar test exists that is designed specifically for misleading/manipulative content (i.e., implied misinformation).…”

“…The Misinformation Susceptibility Test (MIST) Maertens, Götz, et al (2023) developed the MIST with the purpose of creating a psychometrically validated measure of someone's ability to discern true news from false news. For this purpose, they collected 104 "real news" headlines and generated 309 "fake news" headlines.…”

Mapping verification behaviors in the post-truth era: A systematic review

“…In addition, Siapera (2022) criticized that misinformation-suppression tools used by social media platforms help sustain the platform, but cannot solve the problem of misinformation production and dissemination. Researchers have developed algorithms and models to detect online misinformation (Ahmed et al, 2017;Horne and Adali, 2017;Maertens et al, 2023;Yang et al, 2019;Zhou et al, 2020;Zhou and Zafarani, 2019). Nevertheless, machine-learning models can only be adapted to specific contexts under most circumstances.…”