Differential Item Functioning Assessment in Cognitive Diagnostic Modeling: Application of the Wald Test to Investigate DIF in the DINA Model

Abstract

Analyzing examinees' responses using cognitive diagnostic models (CDMs) has the advantage of providing diagnostic information. To ensure the validity of the results from these models, differential item functioning (DIF) in CDMs needs to be investigated. In this article, the Wald test is proposed to examine DIF in the context of CDMs. This study explored the effectiveness of the Wald test in detecting both uniform and nonuniform DIF in the DINA model through a simulation study. Results of this study suggest that for relatively discriminating items, the Wald test had Type I error rates close to the nominal level. Moreover, its viability was underscored by the medium to high power rates for most investigated DIF types when DIF size was large. Furthermore, the performance of the Wald test in detecting uniform DIF was compared to that of the traditional Mantel-Haenszel (MH) and SIBTEST procedures. The results of the comparison study showed that the Wald test was comparable to or outperformed the MH and SIBTEST procedures. Finally, the strengths and limitations of the proposed method and suggestions for future studies are discussed. 98

Assessment of Differential Item Functioning Under Cognitive Diagnosis Models: The DINA Model Example

“…In the context of CDMs, DIF occurs when examinees with the same latent attribute profile but from different groups (e.g., gender or ethnicity) have different probabilities of answering an item correctly. DIF assessment under CDMs provides evidence for invariance of attribute‐item interactions across groups (Hou, de la Torre, & Nandakumar, ). Estimation of item parameters and person attribute profiles will be contaminated when DIF items exist.…”

“…Given that CDMs have become popular and DIF assessment is essential for test fairness, DIF assessment within the CDM framework is warranted. To our best knowledge, there are only a few studies that investigate DIF within the CDM framework (Hou et al., ; Li, ; Zhang, ). Zhang () proposed the use of estimated attribute profiles as a matching variable for the MH and SIBTEST methods to assess DIF under the DINA model.…”

“…Hou et al () applied the Wald statistic to detect both uniform and nonuniform DIF in the DINA model. In particular, the DINA model was fit to the data of the reference and focal groups separately, and the difference in the item parameter estimates of the studied item was examined with the Wald statistic.…”

An Iterative Method for Empirically-Based Q-Matrix Validation

“…The latter values were just approximations based on the conditions defined in the simulation study. Further discussions about multiple strategies in cognitive diagnosis using the fraction subtraction data can be found in de la Torre and Douglas (2008), Hou andMislevy (1996). Other reasons could be because the fraction subtraction data have fewer number of items and attributes than the simulation study.…”

Fuzzy Cognitive Diagnosis for Modelling Examinee Performance

“…DINA describes an examinee by a latent binary vector which denotes whether she has mastered the skills required by the problem, and a given Q-matrix is used to guarantee the interpretation of the diagnosis results. The DINA-based models are applied to further specific educational scenarios, such as differential item functioning assessment [Hou et al 2014], learning team formation [Liu et al 2016] and comprehension test validation and difficulty estimation across multiple forms [Clark 2013;. Though discrete CDMs are interpretable, their diagnosis results are usually not accurate enough.…”