Differential Item Functioning Detection Across Two Methods of Defining Group Comparisons

Abstract

This study compares two methods of defining groups for the detection of differential item functioning (DIF): (a) pairwise comparisons and (b) composite group comparisons. We aim to emphasize and empirically support the notion that the choice of pairwise versus composite group definitions in DIF is a reflection of how one defines fairness in DIF studies. In this study, a simulation was conducted based on data from a 60-item ACT Mathematics test (ACT; Hanson & Béguin). The unsigned area measure method (Raju) was used as the DIF detection method. An application to operational data was also completed in the study, as well as a comparison of observed Type I error rates and false discovery rates across the two methods of defining groups. Results indicate that the amount of flagged DIF or interpretations about DIF in all conditions were not the same across the two methods, and there may be some benefits to using composite group approaches. The results are discussed in connection to differing definitions of fairness. Recommendations for practice are made.

Detecting Differential Item Functioning: Item Response Theory Methods Versus the Mantel-Haenszel Procedure

“…That is, if the item discrimination parameter had been chosen to be fixed at another value such as 0.8 or 1.2 it would have been more difficult to compare findings based on the IRT model due to the misalignment of item discrimination. In addition, this enhanced the generalizability of findings as data were generated from a different set of parameters each time as opposed to generating item response data based on a single test (Cohen et al, 1996;Sari & Huggins, 2015;Wang & Yeh, 2003). As previously noted, the number of items was fixed at 50 and no DIF items were simulated.…”

“…Another search using the terms "misspecification" and "item response theory" in the title returned only 2 results, neither related to DIF. The existing MC studies of DIF which have examined IRT and non-IRT DIF methods offer varied and sometimes conflicting research recommendations (Cohen et al, 1996;Cohen & Kim, 1993;DeMars, 2010;Kim et al, 1994;Herrera & Gómez, 2008;Lautenschlager & Park, 1988;Li et al, 2012;Lim & Drasgow, 1990;McLaughlin & Drasgow, 1987;Paek, 2010;Rudner et al, 1980;Sari & Huggins, 2015;Shepard et al, 1985;Wang & Yeh, 2003;Wells et al, 2009). Therefore, there still remain unknown aspects regarding these DIF methods such as IRT model fit, IRT model specification and misspecification, sample size, item discrimination variability, and item impact, which are addressed in this study and fill in the gap identified by Raju et al (1993).…”

“…Sample size has varied widely in the MC DIF literature, ranging from 250 to 20,000. To obtain an accurate measure of the item parameters for both the 1PL and 2PL a sample size of 500 is adequate (Holland & Wainer, 1993;Sari & Huggins, 2015). For all three DIF methods the research literature suggests different and sometimes conflicting findings regarding the relationship between sample size and Type I error.…”

Fairness and Comparability in Achievement Motivation Items: A Differential Item Functioning Analysis

“…From a statistical perspective, when comparing two groups, which group serves as the reference versus focal group is arbitrary (Ellis & Kimmel, 1992; Oshima et al, 2015). However, with a multicategory grouping variable such as ethnicity that lends itself to a multiple-group DIF analysis, the choice of reference group becomes more important (Ellis & Kimmel, 1992; Sari & Huggins, 2015). In particular, even if done inadvertently, researchers are potentially making an underlying values statement about standards for comparisons when they select a specific group as the reference group (e.g., using White students as the reference group implies that White students should be the reference point; Martinková et al, 2017; Sari & Huggins, 2015).…”

Performances of MIMIC and Logistic Regression Procedures in Detecting DIF

“…In the DIF analyses of the data, Mplus 6.12 (Muthén & Muthén, 1998 program was used for the MIMIC method and SAS 9.1.3 (SAS Institute, 2007) program was used for the logistic regression method. The DIF analyses were conducted using a pairwise approach in which the groups are compared with each other (i.e., focal group compared with reference group) (Sari & Huggins, 2014).…”