It Might Not Make a Big DIF

Abstract

Differential test functioning, or DTF, occurs when one or more items in a test demonstrate differential item functioning (DIF) and the aggregate of these effects are witnessed at the test level. In many applications, DTF can be more important than DIF when the overall effects of DIF at the test level can be quantified. However, optimal statistical methodology for detecting and understanding DTF has not been developed. This article proposes improved DTF statistics that properly account for sampling variability in item parameter estimates while avoiding the necessity of predicting provisional latent trait estimates to create two-step approximations. The properties of the DTF statistics were examined with two Monte Carlo simulation studies using dichotomous and polytomous IRT models. The simulation results revealed that the improved DTF statistics obtained optimal and consistent statistical properties, such as obtaining consistent Type I error rates. Next, an empirical analysis demonstrated the application of the proposed methodology. Applied settings where the DTF statistics can be beneficial are suggested and future DTF research areas are proposed.

Multiple Group IRT Measurement Invariance Analysis of the Forms of Self-Criticising/Attacking and Self-Reassuring Scale in Thirteen International Samples

“…Nontrivial DTF can occur in the case when the parameters systematically favour one group over another. Consequently, the aggregate of these small, nonsignificant differences at the item-level can become substantial at the test level (Chalmers et al 2016). DTF is more relevant for our purpose than DIF; we do not intend to inspect particular items on FSCRS subscales nor do we intend to improve them.…”

“…Testing the DTF involves two statistical measures (Chalmers et al 2016). The first, the signed DTF tests whether there is any systematic scoring bias indicating that some groups consistently score higher across a specified range of the latent trait, and the second, the unsigned DTF, assess whether the test curves (plots of expected total score against a latent trait) have a large degree of overall separation on average, suggesting that there may be substantial DTF at particular levels of latent trait.…”

Numerical approximation of the observed information matrix with Oakes' identity

“…In statistical applications where the quality of every element in the ACOV is important (e.g. Chalmers, 2017b;Chalmers, Counsell, & Flora, 2016;Woods et al, 2012), obtaining an accurate and reliable representation for all elements of the ACOV matrix is essential so that any systematic effects borne of numerical inaccuracies can faithfully be avoided.…”

Sense of school belonging: Psychometric properties and differences across gender, grades, and East Asian societies

“…To determine whether items are influenced by person characteristics or context (item fairness), differential item functioning (DIF) was investigated for genders, grades (Grade 4 and Grade 8), and societies (Chinese Taipei, Hong Kong, Japan, and South Korea). DIF occurs when one subgroup of a population shows a different probability of endorsing an item than another subgroup when they have the same ability/inclination (Chalmers, Counsell, & Flora, 2016). The presence of DIF implies that the scale is not fair.…”