The Dire Disregard of Measurement Invariance Testing in Psychological Science

Abstract

In psychological science, self-report scales are widely used to compare means in targeted latent constructs across time points, groups, or experimental conditions. For these scale mean comparisons (SMC) to be meaningful and unbiased, the scales should be measurement invariant across the compared time points or (experimental) groups. Measurement invariance (MI) testing checks whether the latent constructs are measured equivalently across groups or time points. Since MI is essential for meaningful comparisons, we conducted a systematic review to check whether MI is taken seriously in psychological research. Specifically, we sampled 426 psychology articles with openly available data that involved a total of 918 SMCs to (1) investigate common practices in conducting and reporting of MI testing, (2) check whether reported MI test results can be reproduced, and (3) conduct MI tests for the SMCs that enabled sufficiently powerful MI testing with the shared data. Our results indicate that (1) 4% of the 918 scales underwent MI testing across groups or time and that these tests were generally poorly reported, (2) none of the reported MI tests could be successfully reproduced, and (3) of 161 newly performed MI tests, a mere 46 (29%) reached sufficient MI (scalar invariance), and MI often failed completely (89; 55%). Thus, MI tests were rarely done and poorly reported in psychological studies, and the frequent violations of MI indicate that reported group differences cannot be solely attributed to group differences in the latent constructs. We offer recommendations on reporting MI tests and improving computational reproducibility practices.

Three-step latent class analysis with inverse propensity weighting in the presence of differential item functioning

“…Note, that regardless of the definition of the ATE, DIF always needs to be accounted for when present as the estimate of the ATE will otherwise be biased. Unfortunately, testing for DIF seems to be an uncommon practice (D'Urso et al, 2022).…”

Severity and Crises in Science: Are We Getting It Right When We're Right and Wrong When We're Wrong?

“…Alongside the increased concerns about the quality of statistical analyses and theories, scientists have raised awareness of a measurement and validity crisis in psychology (D'Urso, Maassen, van Assen, Nuijten, Roover, & Wicherts, 2022, Kane et al, 2021, Flake & Fried, 2020, Schimmack, 2021Fabrigar, Wegener, & Petty, 2020, Vazire, Schiavone, & Bottesini, 2022. In response to the replication crisis, Fabrigar and Wegner (2015) suggest that failures to replicate can sometimes be caused by a lack of measurement invariance.…”

Prácticas Cuestionables en Estudios de Validez de Instrumentos de Medición Psicológica: Comunalidades y Unicidades de la Crisis de Replicabilidad en el Campo de la Psicometría

“…No solo hay problemas de calidad en las investigaciones donde se usan AFE, también existen en investigaciones que analizan la invarianza factorial (D'Urso et al, 2022). Las conclusiones de este estudio apuntan a la necesidad de mejorar la forma de comunicar los resultados, ya que esto es vital para la reproducibilidad y replicabilidad (Nosek et al, 2022).…”