Describing and Categorizing Dif in Polytomous Items

Abstract

The purpose of this project was to evaluate statistical procedures for assessing differential item functioning (DIF) in polytomous items (items with more than two score categories). Three descriptive statistics—the Standardized Mean Difference, or SMD (Dorans & Schmitt, 1991), and two procedures based on SIBTEST (Shealy & Stout, 1993) were considered, along with five inferential procedures—two based on SMD, two based on SIBTEST, and the Mantel (1963) method. The DIF procedures were evaluated through applications to simulated data, as well as data from ETS tests. The simulation included conditions in which the two groups of examinees had the same ability distribution and conditions in which the group means differed by one standard deviation. When the two groups had the same distribution, the descriptive index that performed best was the SMD. When the two groups had different distributions, a modified form of the SIBTEST DIF effect size measure tended to perform best. The five inferential procedures performed almost indistinguishably when the two groups had identical distributions. When the two groups had different distributions and the studied item was highly discriminating, the SIBTEST procedures showed much better Type I error control than did the SMD and Mantel methods, particularly in short tests. The power ranking of the five procedures was inconsistent; it depended on the direction of DIF and other factors. Routine application of these polytomous DIF methods at ETS seems feasible in cases where a reliable test is available for matching examinees. For the Mantel and SMD methods, Type I error control may be a concern under certain conditions. In the case of SIBTEST, the current version cannot easily accommodate matching tests that do not use number‐right scoring. Additional research in these areas is likely to be useful.

Sense of school belonging: Psychometric properties and differences across gender, grades, and East Asian societies

“…This study adopted absolute DIF contrast sizes greater than .64 as the threshold for biased items across groups. This threshold has been used in high-stakes tests, such as the Graduate Record Examination (Zwick, Thayer, & Mazzeo, 1997); the Test of English as a Foreign Language (Young, Morgan, Rybinski, Steinberg, & Wang, 2013); other language tests, such as the TestDaF language test in German (Eckes, 2013); and empirical research (Chen & Zhu, 2017).…”

A Review of Developments and Applications in Item Analysis

“…2.16 have been considered for polytomously scored items by ETS researchers, including Dorans and Schmitt (1993), Moses et al (2013), and Zwick et al (1997). At the time of this writing, there is great interest in developing more innovative items that utilize computer delivery and are more interactive in how they engage examinees.…”

Constructed‐response Dif Evaluations for Mixed‐format Tests

“…1 Evaluations of differential item functioning (DIF) in constructed-response (CR) items have been fairly limited in ETS testing programs in spite of considerable research attention (Chang, Mazzeo, & Roussos, 1996;Dorans & Schmitt, 1993;Kim, Cohen, Alagoz, & Kim, 2007;Kristjansson, Aylesworth, McDowell, & Zumbo, 2005;Penfield, 2007;Penfield & Algina, 2006;Zwick, Donoghue, & Grima, 1993;Zwick, Thayer, & Mazzeo, 1997). A statement made in a study conducted more than 15 years ago is an accurate description of current CR DIF practice: "At present, ETS has no official policy for screening polytomous items for DIF" (Zwick et al, 1997, p. 1).…”