Detecting Differential Item Functioning Using Multiple-Group Cognitive Diagnosis Models

Abstract

This study proposes a multiple-group cognitive diagnosis model to account for the fact that students in different groups may use distinct attributes or use the same attributes but in different manners (e.g., conjunctive, disjunctive, and compensatory) to solve problems. Based on the proposed model, this study systematically investigates the performance of the likelihood ratio (LR) test and Wald test in detecting differential item functioning (DIF). A forward anchor item search procedure was also proposed to identify a set of anchor items with invariant item parameters across groups. Results showed that the LR and Wald tests with the forward anchor item search algorithm produced better calibrated Type I error rates than the ordinary LR and Wald tests, especially when items were of low quality. A set of real data were also analyzed to illustrate the use of these DIF detection procedures.

Don’t worry about the anchor-item setting in longitudinal learning diagnostic assessments

“…Previous researches which conducted in CTT and IRT have found that a certain percentage of anchor items are needed; however, in LDA, the attributes on different occasions are on the same scale because the interpretation of attribute is deterministic ( Madison and Bradshaw, 2018 ; Ma et al, 2021 ). This essential difference leads to attribute estimates at different occasions that are naturally on a same scale, without the need to build linking between them through anchor items.…”

“…However, a recent study of the hidden Markov model-based longitudinal DCM pointed out that no anchor items are necessary in longitudinal LDAs because the scale for DCMs in non-arbitrary (Madison and Bradshaw, 2018) or the interpretation of attribute is deterministic (Ma et al, 2021). The results of Madison and Bradshaw 1 Also known as the reachability matrix (Tatsuoka, 1986) for structurally independent attributes, which specifies the direct and indirect relationships among the attributes.…”

“…However, a recent study of the hidden Markov model-based longitudinal DCM pointed out that no anchor items are necessary in longitudinal LDAs because the scale for DCMs in non-arbitrary ( Madison and Bradshaw, 2018 ) or the interpretation of attribute is deterministic ( Ma et al, 2021 ). The results of Madison and Bradshaw (2018) study indicated that even in tests without anchor items, the hidden Markov model-based longitudinal DCM can classify respondents accurately and reliably.…”

Comparison of Methods Used in Detection of DIF in Cognitive Diagnostic Models with Traditional Methods: Applications in TIMSS 2011

“…It is stated in the literature that many important statistical routines are needed to ensure appropriate uses and interpretations and to unlock the potentials of CDMs, such as the procedure for detecting differential item functioning (DIF), which can be used to determine item parameter bias (Ma et al, 2021;Paulsen et al, 2020). DIF has been described traditionally as "the probability of students with the same total test score or ability level but in different groups to correctly respond to an item when the variable is unrelated to the construct of interest" (Hou et al, 2014).…”

Impact of Retrofitting and Item Ordering on DIF

“…These techniques include Mantel-Haenszel (Holland & Thayer, 1988), Logistic Regression (Swaminathan & Rogers, 1990), IRTLR tests (Thissen & Steinberg, 1988), Lord's χ2 test (1980), and the MIMIC model (Jӧreskog & Goldberger, 1975;Woods, 2009). Recently, DIF detection techniques for cognitive diagnosis modeling framework have also been proposed (Hou, Terzi & de la Torre, 2020;Ma, Terzi & de la Torre, 2021). For example, Hou, de la Torre, and Nandakumar (2014) proposed a DIF detection method based on the Wald test that is compatible with the deterministic inputs, noisy "and" gate (DINA: Junker & Sijtsma, 2001) model.…”