Effects of Differential Item Discriminations between Individual-Level and Cluster-Level under the Multilevel Item Response Theory Model

Abstract

This study attempted to interpret differential item discriminations between individual and cluster levels by focusing on patterns and magnitudes of item discriminations under 2PL multilevel IRT model through a set of variety simulation conditions. The consistency between the mean of individual-level ability estimates and cluster-level ability estimates was evaluated by the correlations between them. As a result, it was found that they were highly correlated if the patterns of item discriminations were the same for both individual and cluster levels. The magnitudes of item discriminations themselves did not affect much on correlations, as far as the patterns were the same at the two levels. However, it was found that the correlation became lower when the patterns of item discriminations were different between the individual and cluster levels. Also, it was revealed that the mean of the estimated individual-level abilities would not be necessarily a good representation of the cluster-level ability, if the patterns were different at the two levels.

Detecting Differential Item Discrimination (DID) and the Consequences of Ignoring DID in Multilevel Item Response Models

“…The simulation conditions include the number of clusters, cluster size (i.e., the number of individuals per cluster) and design (i.e., balanced vs. unbalanced), and the ICC. Further, simulation conditions related to DID, such as the DID magnitudes and the number of items that have DID, were also considered (e.g., Patarapichayatham & Kamata, ). In this study, the ICC was chosen as a simulation condition instead of DID magnitudes because the degree of the ICC reflects the DID magnitudes (as shown in Equation ).…”

“…As examples, in educational research, within‐class averages of student achievement or socioeconomic status can be used to create a classroom contextual variable (Marsh et al., ). In the presence of this kind of between‐level construct, the mean of the within‐level scores is approximately equal to the between‐level scores (Patarapichayatham & Kamata, ).…”

“…Testing the assumption that the item discriminations are the same over levels, called cross‐level invariance in terms of item discriminations (Marsh et al., ), is important in real data applications. The violation of the cross‐level invariance is called differential item discrimination (DID; Patarapichayatham & Kamata, ). However, cross‐level invariance is assumed implicitly in many multilevel item response model applications.…”

“…De Jong, Steenkamp, and Fox () presented random item response models to test whether individual‐level item parameters differ over clusters. Patarapichayatham and Kamata () showed the effects of different patterns and magnitudes of item discriminations over levels on estimates of within‐level and between‐level latent variables in a two‐parameter multilevel item response model. However, there is a lack of research on the evaluation of DID detection methods and the consequences of ignoring DID in terms of the accuracy of parameter estimates in the use of multilevel item response models.…”