Impact of Retrofitting and Item Ordering on DIF

Abstract

Richer diagnostic information about examinees' cognitive strength and weaknesses are obtained from cognitively diagnostic assessments (CDA) when a proper cognitive diagnosis model (CDM) is used for response data analysis. To do so, researchers state that a preset cognitive model specifying the underlying hypotheses about response data structure is needed. However, many real data CDM applications are adds-on to simulation studies and retrofitted to data obtained from non-CDAs. Such a procedure is referred to as retrofitting, and fitting CDMs to traditional test data is not uncommon. To deal with a major validity concern of item/test bias in CDAs, some recent DIF detection techniques compatible with various CDMs have been proposed. This study employs several DIF detection techniques developed based on CTT, IRT, and CDM frameworks and compares the results to understand the extent to which DIF flagging behavior of items is affected by retrofitting. A secondary purpose of this study is to gather evidence about test booklet effects (i.e., item ordering) on items' psychometric properties through DIF analyses. Results indicated severe DIF flagging prevalence differences for items across DIF detection techniques employing Wald test, Raju's area measures, and Mantel-Haenzsel statistics. The largest numbers of DIF cases were observed when the data were retrofitted to a CDM. The results further revealed that an item might be flagged as DIF in one booklet, whereas it might not be flagged in another.

Differential item functioning detection in fundamental physics test

“…If the item parameters and the ICC focal and reference group are identical, the area between the ICC will be 0. But, if the area is more than zero, the item may contain DIF (Akbay, 2021). After knowing the Model's fit, the first step of this method is estimating parameters, discriminant index (a), and difficulty level (b) parameters; of focal and reference groups.…”

Assessing the fairness of mathematical literacy test in Indonesia: Evidence from gender-based differential item function analysis

“…One year later, Hou et al (2020) employed the Wald test equation to determine DIF in the CDM. Additionally, Akbay (2021) utilized three methods for assessing DIF, namely, the Classical Test Theory (MH) approach, Item Response Theory (Raju), and CDM (Wald Test), to investigate the psychometric attributes of the test. Large-scale assessment data were used to observe DIF determination patterns using the three DIF detection methods.…”

Comparison of Methods Used in Detection of DIF in Cognitive Diagnostic Models with Traditional Methods: Applications in TIMSS 2011

“…This information, along with feedback opportunities for teachers and programs, provides students with opportunities for individualized learning support that compensates for learning deficiencies. In addition to this contribution, DIF detection, one of the most important statistical routines for ensuring proper usage and interpretation, appears to be a helpful method, mainly when dealing with the issue of validity, which is a problem with traditional methods (Akbay, 2021). Therefore, detecting DIF has become a standard procedure in psychometric analyses.…”

“…The performances of the items in the real dataset were investigated under various simulations, and the compatibility of the attributes' classifications was evaluated when saturated and reduced models were used. In the CTT, IRT, and CDM framework, Akbay (2021) investigated the test's psychometric properties using DIF determination methods (i.e., MH, Raju area measures, and Wald test for DIF). DIF flagging patterns of three different DIF detection methods were observed when real data from a largescale assessment (TEOG) were retrofitted.…”