Extensions of Mantel–Haenszel for Multilevel DIF Detection

Abstract

Multilevel data structures are ubiquitous in the assessment of differential item functioning (DIF), particularly in large-scale testing programs. There are a handful of DIF procures for researchers to select from that appropriately account for multilevel data structures. However, little, if any, work has been completed to extend a popular DIF method to this case. Thus, the primary goal of this study was to introduce and investigate the effectiveness of several new options for DIF assessment in the presence of multilevel data with the Mantel–Haenszel (MH) procedure, a popular, flexible, and effective tool for DIF detection. The performance of these new methods was compared with the standard MH technique through a simulation study, where data were simulated in a multilevel framework, corresponding to examinees nested in schools, for example. The standard MH test for DIF detection was employed, along with several multilevel extensions of MH. Results demonstrated that these multilevel tests proved to be preferable to standard MH in a wide variety of cases where multilevel data were present, particularly when the intraclass correlation was relatively large. Implications of this study for practice and future research are discussed.

Assessment of Differential Item Functioning in Health-Related Outcomes: A Simulation and Empirical Analysis with Hierarchical Polytomous Data

“…This selected range is in accord with range of ICC that is seen in applied educational and recent simulation studies [6, 20, 22, 23]. …”

“…It is important to ensure that latent traits of all examinees were accurately determined by items and test scores. Although many studies have been done to expand DIF detection methods to polytomously scored items [23] and great attention also has been given to multilevel ordinal logistic regression, relatively few of them have focused on hierarchical polytomous DIF that is often present in a variety of psychological, medical, and educational researches.…”

“…For example, Kamata et al (2005) used generalized Rasch model with group membership to investigate detection of DIF [21]. In a series of studies, French and Finch expanded logistic regression, multiple indicator multiple cause (MIMIC), and Mantel–Haenszel models for detecting DIF in multilevel data [6, 22, 23]. Recently, Jin et al (2013) studied the effect of item's and total score's intraclass correlation coefficient (ICC) on DIF, and Wen (2014) investigated DIF detection in both levels simultaneously.…”

Measurement invariance techniques to enhance measurement sensitivity

“…This method has been shown to work with detecting DIF (French and Finch, 2013). We employed the delta statistic as an effect size measure and ETS classification (e.g., A, B, C) to judge the magnitude of DIF and supplement the chi-square significance test.…”

Comparing DIF methods for data with dual dependency

“…Previous studies, however, have investigated person and item clustering effects on the comparative performance of several DIF methods, separately (e.g., French and Finch 2013;Wang and Wilson 2005).…”