Detecting Differential Item Functioning Using Posterior Predictive Model Checking: A Comparison of Discrepancy Statistics

Abstract

This study proposes a new Bayesian differential item functioning (DIF) detection method using posterior predictive model checking (PPMC). Item fit measures including infit, outfit, observed score distribution (OSD), and Q1 were considered as discrepancy statistics for the PPMC DIF methods. The performance of the PPMC DIF method was evaluated via a Monte Carlo simulation manipulating sample size, DIF size, DIF type, DIF percentage, and subpopulation trait distribution. Parametric DIF methods, such as Lord's chi-square and Raju's area approaches, were also included in the simulation design in order to compare the performance of the proposed PPMC DIF methods to those previously existing. Based on Type I error and power analysis, we found that PPMC DIF methods showed better-controlled Type I error rates than the existing methods and comparable power to detect uniform DIF. The implications and recommendations for applied researchers are discussed.

Alternatives to Weighted Item Fit Statistics for Establishing Measurement Invariance in Many Groups

“…Although we used the cutoff of .12 to be practical and consistent with the operational scaling procedure, it is worthwhile to evaluate the effectiveness of various cutoff values for the alternative RMSD statistics. Furthermore, alternative DIF detection methods other than the RMSD statistic can also be investigated in the context of international large-scale assessments (e.g., Joo & Lee, 2022; Joo, Lee, & Stark, 2022).…”

“…Furthermore, we limited the generated data to dichotomous responses and uniform DIF to keep the study manageable and to avoid confounding factors. We investigated uniform DIF because previous studies reported that uniform DIF is more prevalent in operational settings (Joo et al, 2022; Joo & Lee, 2022) and the RMSD is more sensitive to uniform DIF than nonuniform DIF (Buchholz & Hartig, 2019).…”

“…Thus, we examined 5% (small), 15% (medium), and 30% (large) of DIF items, which resulted in 2, 6, or 12 items being chosen as DIF items. We expect that as the percentage of DIF items increases, the power to detect DIF using the original RMSD statistic would decrease due to the equality constraint across all items in the multigroup IRT model estimation (Buchholz & Hartig, 2019; Joo et al, 2021; Joo & Lee, 2022; von Davier et al, 2019). Consequently, as more items are contaminated with DIF, more item parameter estimates could be biased and DIF detection would decrease.…”

Detecting Multidimensional DIF in Polytomous Items with IRT Methods and Estimation Approaches

“…These approaches are generally multidimensional extensions of traditional DIF detection approaches developed for unidimensional tests (e.g., multidimensional SIBTEST (Stout et al, 1987), the IRT likelihood ratio test (IRT-LR; Thissen, et al, 1993;Suh & Cho, 2014), and Multiple Indicators Multiple Causes Model (MIMIC; Lee et al, 2017). However, relatively new multidimensional DIF estimation statistics that provide different estimation algorithms as an alternative to traditional DIF detection methods, have also been developed (e.g., Yuan & Lin, 2006;Bauer et al, 2020;Wang et al, 2020;Joo & Lee, 2022;Lim et al, 2022).…”

A Unified Comparison of IRT‐Based Effect Sizes for DIF Investigations

“…The study of differential item functioning (DIF) has a long history in the psychometric literature, dating back to the pioneering work of several psychometricians involved with the Educational Testing Service (ETS; e.g., Donlon, 1981; Dorans & Kulick, 1986), and continues to receive considerable methodological attention (e.g., Joo & Lee, 2022; Wells, 2021). Procedures for detecting DIF are generally well established, and can utilize various strategies based on observed variable and latent variable methodologies (see Millsap, 2011, for an in‐depth presentation).…”