Examining DIF in the Context of CDMs When the Q-Matrix Is Misspecified

Abstract

The rise in popularity and use of cognitive diagnostic models (CDMs) in educational research are partly motivated by the models’ ability to provide diagnostic information regarding students’ strengths and weaknesses in a variety of content areas. An important step to ensure appropriate interpretations from CDMs is to investigate differential item functioning (DIF). To this end, the current simulation study examined the performance of three methods to detect DIF in CDMs, with particular emphasis on the impact of Q-matrix misspecification on methods’ performance. Results illustrated that logistic regression and Mantel–Haenszel had better control of Type I error than the Wald test; however, high power rates were found using logistic regression and Wald methods, only. In addition to the tradeoff between Type I error control and acceptable power, our results suggested that Q-matrix complexity and item structures yield different results for different methods, presenting a more complex picture of the methods’ performance. Finally, implications and future directions are discussed.

A Comparison of Differential Item Functioning Detection Methods in Cognitive Diagnostic Models

“…In general, the empirical Type I error rates for W XPD , W Obs , and W Sw were better than those for W d under all conditions. Consistent with the results reported in previous studies (e.g., Hou et al, 2014; Wang et al, 2014; Svetina et al, 2018), the Type I error rates for W d were somewhat inflated under most of the conditions. Moreover, although the performances of W XPD , W Obs , and W Sw seemed to be influenced by the attribute correlation, the Type I error rates for those methods were reasonably close to the nominal Type I rate of 0.05 under most of the simulation conditions.…”

“…Under the null hypothesis of H 0 : Cv j = 0 , W d asymptotically follows a chi-square distribution with 2 degrees of freedom. However, authors of previous studies (Hou et al, 2014; Svetina et al, 2018) showed that W d tended to have inflated Type I errors and the W XPD and W Obs performed better than that for W d with regard to Type I error control (Liu et al, 2016b).…”

“…The purpose of this simulation study was to systematically investigate the Type I error and power performances of W d , W XPD , W Obs , W Sw , MH, and LR for detecting DIF. The settings of the simulation draw on those of previous real data analyses and simulations on DIF detection methods in CDMs (e.g., de la Torre and Douglas, 2004; Hou et al, 2014; Li and Wang, 2015; Svetina et al, 2018). The test length, sample size, and number of attributes were fixed to J = 30, N = 1, 000, and K = 5, respectively, and the binary item response data sets were generated from the DINA model.…”

“…Li and Wang (2015) compared the empirical performance of the LCDM-DIF method with the Wald method for two and three groups using the MCMC algorithm, and they found that the Type I error rates of the LCDM-DIF were better controlled than the Wald statistic under most conditions, however, for the three-group conditions, the power of the Wald method was slightly better than that of the LCDM-DIF. Svetina et al (2018) evaluated the impact of Q-matrix misspecification on the performance of LR, MH, and W d for detecting DIF in CDMs. They found that the Type I error rate control of LR and MH was better than that of W d ; LR and W d had greater power than MH and the performance of LR, MH, and W d was affected by Q-matrix misspecification.…”

“…The W d test for DIF detection that was used in previous studies (Hou et al, 2014; Svetina et al, 2018) was based on item-wise information matrix. However, Liu et al (2016a) pointed out that because the item and the structural model parameters are simultaneously estimated from the item response data, the information matrix of CDMs should contain both item and structural parameters.…”