Alternatives to Weighted Item Fit Statistics for Establishing Measurement Invariance in Many Groups

Abstract

Evaluating scale comparability in international large-scale assessments depends on measurement invariance (MI). The root mean square deviation (RMSD) is a standard method for establishing MI in several programs, such as the Programme for International Student Assessment and the Programme for the International Assessment of Adult Competencies. Previous research showed that the RMSD was unable to detect departures from MI when the latent trait distribution was far from item difficulty. In this study, we developed three alternative approaches to the original RMSD: equal, item information, and b-norm weighted RMSDs. Specifically, we considered the item-centered normalized weight distributions to compute the item characteristic curve difference in the RMSD procedure more efficiently. We further compared all methods’ performance via a simulation study and the item information and b-norm weighted RMSDs showed the most promising results. An empirical example is demonstrated, and implications for researchers are discussed.

Detecting differential item functioning in presence of multilevel data: do methods accounting for multilevel data structure make a DIFference?

“…We considered uniform DIF only. While we recognize that nonuniform DIF is also possible (see "5 Discussion") our choice to only examine uniform DIF was driven by several factors, including that uniform DIF has been more prevalent in operational settings in some contexts (e.g., Joo et al, 2023), that it would allow us to examine commonly used methods (e.g., single-level MH), and lastly, to keep our study manageable. 4…”