Detecting DIF in Multidimensional Forced Choice Measures Using the Thurstonian Item Response Theory Model

Abstract

Although modern item response theory (IRT) methods of test construction and scoring have overcome ipsativity problems historically associated with multidimensional forced choice (MFC) formats, there has been little research on MFC differential item functioning (DIF) detection, where item refers to a block, or group, of statements presented for an examinee’s consideration. This research investigated DIF detection with three-alternative MFC items based on the Thurstonian IRT (TIRT) model, using omnibus Wald tests on loadings and thresholds. We examined constrained and free baseline model comparisons strategies with different types and magnitudes of DIF, latent trait correlations, sample sizes, and levels of impact in an extensive Monte Carlo study. Results indicated the free baseline strategy was highly effective in detecting DIF, with power approaching 1.0 in the large sample size and large magnitude of DIF conditions, and similar effectiveness in the impact and no-impact conditions. This research also included an empirical example to demonstrate the viability of the best performing method with real examinees and showed how a DIF and a DTF effect size measure can be used to assess the practical significance of MFC DIF findings.

A Review of Measurement Equivalence in Organizational Research: What's Old, What's New, What's Next?

“…Finally, all items can be tested for DIF because they are on a common metric. This approach appears flexible as it was recently extended for use on multidimensional forced-choice measures (Lee et al 2021) and has been recommended for use in IRT whenever equivalent anchor items are unknown (Tay et al 2015). However, although these approaches have primarily been used in the IRT literature, they can also be generalized to the CFA framework.…”

Detecting Differential Item Functioning Using Posterior Predictive Model Checking: A Comparison of Discrepancy Statistics

“…For example, in Programme for International Student Assessment (PISA) 2018 reading assessment, the item parameter differences for the discrimination and difficulty parameters across the participating countries ranged from .01 to .88 and from .05 to 1.17, respectively (Joo et al., 2021; OECD, 2019). In addition, we also considered the 10% and 20% DIF conditions because the proportions are commonly found in practice (e.g., Joo et al., 2021; Lee et al., 2021; Stark et al., 2004, 2006) and research settings (e.g., Kim & Cohen, 1992, 1998; Oshima et al., 1997, 2006; Rutkowski & Svetina, 2014).…”

Multidimensional IRT for forced choice tests: A literature review

“…DIF is the parameter inconsistency at the item level. P. Lee et al [ 70 ] proposed an Omnibus Wald test for the discrimination and intercept indicators of the TIRT and suggested through simulation research that the detection efficiency was higher under the free baseline method: the detection rate was close to 1 and the type I error rate was close to 0.05 as sample size and DIF amount increased. Qiu & Wang [ 71 ] proposed three DIF test methods for RIM including EMD (equal-mean-difficulty), AOS (all-other-statement), and CS (constant-statement).…”

“…At present, there is only research on the parameter invariance of TIRT [ 69 , 70 ] and RIM [ 71 ]. Future studies should broaden the repertoire of differential item functioning (DIF) test methods for the forced-choice model and improve their sensitivity in detecting DIF from multiple sources.…”