Assessment of Differential Item Functioning Under Cognitive Diagnosis Models: The DINA Model Example

Abstract

The assessment of differential item functioning (DIF) is routinely conducted to ensure test fairness and validity. Although many DIF assessment methods have been developed in the context of classical test theory and item response theory, they are not applicable for cognitive diagnosis models (CDMs), as the underlying latent attributes of CDMs are multidimensional and binary. This study proposes a very general DIF assessment method in the CDM framework which is applicable for various CDMs, more than two groups of examinees, and multiple grouping variables that are categorical, continuous, observed, or latent. The parameters can be estimated with Markov chain Monte Carlo algorithms implemented in the freeware WinBUGS. Simulation results demonstrated a good parameter recovery and advantages in DIF assessment for the new method over the Wald method.

Validity of the listening module of international English language testing system: multiple sources of evidence

“…DIF exists when different groups of learners have different probability of successfully answering an item (Drabinova and Martinkova 2017;Ferne and Rupp 2007;Li and Wang 2015); therefore, if the test takers have less or more the same knowledge, then they should perform similarly on test items; DIF is needed for test validity and test fairness (Fidalgo et al 2014;Hou et al 2014;Pae 2004Pae , 2012Su and Wang 2005;Zumbo 2003Zumbo , 2007.…”

“…As for the significance of CDM, George and Robitzsch (2014) recommend the use of CDM as one of the recent statistical tools for detecting DIF and plenty of psychometric questions in relation to DIF can be addressed with use of CDM (Hou et al 2014). To date, only a few studies have been conducted on DIF assessment within the framework of CDM (Drabinova and Martinkova 2017;Li and Wang 2015;Hou et al 2014;Li 2008;Zhang 2006). However an extensive body of research has been done in the area of cognitive diagnosis of students' learning (Li and Wang 2015; de la Torre 2011; de la Torre and Douglas 2004;Junker and Sijtsma 2001), no study has so far been done on detecting the DIF of IELTS LCT with use of CDMs, so that some researchers (e.g., George and Robitzsch 2014) suggest the use of CDM for DIF detection.…”

Information matrix estimation procedures for cognitive diagnostic models

“…For the past decade, cognitive diagnostic models (CDMs) as a class of statistical models have been widely researched in educational and psychological measurement (Greeno, 1980;Leighton & Gierl, 2007;Rupp, Templin, & Henson, 2010). The information matrix evaluated at the estimates of the model parameters or its inverse variance-covariance matrix has many important applications in CDMs, such as model comparisons at the item level (de la Torre & Lee, 2013), differential item functioning detection (Hou, de la Torre, & Nandakumar, 2014;Li & Wang, 2015), overall goodness-of-fit statistics (Cai & Hansen, 2013;Liu, Tian, & Xin, 2016;Maydeu-Olivares & Joe, 2005), and the computation of approximate confidence intervals for item parameters.…”

“…In CDMs, two procedures to directly estimate the information matrix associated with the item parameter estimates were proposed by de la Torre (2009). These two information matrices have been widely used in the computation of Wald statistics for model comparison at the item level (de la Torre & Lee, 2013) and for differential item functioning detection (Hou et al, 2014;Li & Wang, 2015). However, the Type I error rates of these Wald statistics were inflated, suggesting that new methods to compute the information matrix in CDMs are needed (Ma, Iaconangelo, & de la Torre, 2016).…”

Obituary

“…He guided each of his PhD students to focus on one idiosyncratic characteristic at a time, identify the issue, and then solve the problem. This dedication yielded impactful new knowledge, including models for randomness in subjective judgements for rating scale items (Wang & Wu, ; Wang et al ., ), testlet items (Huang & Wang, ), various rater errors (Hung & Wang, ; Wang & Liu, ; Wang, Su, & Qiu, ), multilevel data structures (Wang & Qiu, ), higher‐order latent traits (Huang & Wang, ; Huang, Wang, Chen, & Su, ), CDMs (Li & Wang, ), unfolding models for Likert‐type items (Liu & Wang, ; Wang & Wu, ), response styles (Chen, Jin, & Wang, ; Jin & Wang, ; Liu & Wang, in press), ipsative items (Wang et al ., ), examinee‐selected items (Liu & Wang, ), and test‐takers with inattentive response behaviour (Jin, Chen, & Wang, ).…”