Differential Item Functioning Effect Size From the Multigroup Confirmatory Factor Analysis for a Meta-Analysis: A Simulation Study

Abstract

This study presents a new approach to synthesizing differential item functioning (DIF) effect size: First, using correlation matrices from each study, we perform a multigroup confirmatory factor analysis (MGCFA) that examines measurement invariance of a test item between two subgroups (i.e., focal and reference groups). Then we synthesize, across the studies, the differences in the estimated factor loadings between the two subgroups, resulting in a meta-analytic summary of the MGCFA effect sizes (MGCFA-ES). The performance of this new approach was examined using a Monte Carlo simulation, where we created 108 conditions by four factors: (1) three levels of item difficulty, (2) four magnitudes of DIF, (3) three levels of sample size, and (4) three types of correlation matrix (tetrachoric, adjusted Pearson, and Pearson). Results indicate that when MGCFA is fitted to tetrachoric correlation matrices, the meta-analytic summary of the MGCFA-ES performed best in terms of bias and mean square error values, 95% confidence interval coverages, empirical standard errors, Type I error rates, and statistical power; and reasonably well with adjusted Pearson correlation matrices. In addition, when tetrachoric correlation matrices are used, a meta-analytic summary of the MGCFA-ES performed well, particularly, under the condition that a high difficulty item with a large DIF was administered to a large sample size. Our result offers an option for synthesizing the magnitude of DIF on a flagged item across studies in practice.

A cross-sectional study on the impact of the COVID-19 pandemic on psychological outcomes: Multiple indicators and multiple causes modeling

“…DIF depicts that the degree of performance on a given item is systematically varied. For example, the DIF is proved when examinees in subgroups (e.g., race/ethnicity, gender, socioeconomic status) with the same level of latent qualities have different probabilities of responding correctly to a particular item [ 66 ]. Therefore, the MIMIC confirmatory factor analysis was used in this study to examine DIF in polytomous items, which are commonly used in educational assessments (e.g., constructed-response items) and psychological inventories (e.g., Likert-type items and rating scale items) [ 67 ].…”

Generalized Mantel–Haenszel Estimators for Simultaneous Differential Item Functioning Tests

“…Otherwise, without the local item independence assumption, a multiple testing adjustment such as the Bonferroni correction is needed (Kim & Oshima, 2013) to control the type I error. Many researchers (Kim & Oshima, 2013; Magis et al, 2015; Park et al, 2021; Penfield, 2001; Rothman, 1990; Sauder & DeMars, 2020; Thissen et al, 2002) have discussed methods for simultaneous and multiple comparison tests.…”