Consequences of Ignoring Guessing Effects on Measurement Invariance Analysis

Abstract

Pseudo-guessing parameters are present in item response theory applications for many educational assessments. When sample size is not sufficiently large, the guessing parameters may be ignored from the analysis. This study examines the impact of ignoring pseudo-guessing parameters on measurement invariance analysis, specifically, on item difficulty, item discrimination, and mean and variance of ability distribution. Results show that when non-zero guessing parameters are ignored from the measurement invariance analysis, item discrimination estimates tend to decrease particularly for more difficult items, and item difficulty estimates decrease unless the items are highly discriminating and difficult. As the guessing parameter increases, the size of the decrease in item discrimination and difficulty tends to increase, and the estimated mean and variance of ability distribution tend to be inaccurate. When two groups have heterogeneous ability distributions, ignoring the guessing parameter affects the reference group and the focal group differently. Implications of result findings are discussed.

Differential item functioning detection in fundamental physics test

“…As previously mentioned, the concept of item bias or DIF is defined as the between two groups and there is a disparity in their chances of answering properly, usually called the Focal group and the Reference Group (Angoff, 1993, Cuhadar et al, 2021. In unidimensional IRT, DIF is expressed as the difference in correctly answering an item among the focal and the reference groups.…”