Detecting Differential Item Discrimination (DID) and the Consequences of Ignoring DID in Multilevel Item Response Models

Abstract

Cross-level invariance in a multilevel item response model can be investigated by testing whether the within-level item discriminations are equal to the between-level item discriminations. Testing the cross-level invariance assumption is important to understand constructs in multilevel data. However, in most multilevel item response model applications, the cross-level invariance is assumed without testing of the cross-level invariance assumption. In this study, the detection methods of differential item discrimination (DID) over levels and the consequences of ignoring DID are illustrated and discussed with the use of multilevel item response models. Simulation results showed that the likelihood ratio test (LRT) performed well in detecting global DID at the test level when some portion of the items exhibited DID. At the item level, the Akaike information criterion (AIC), the sample-size adjusted Bayesian information criterion (saBIC), LRT, and Wald test showed a satisfactory rejection rate (> .8) when some portion of the items exhibited DID and the items had lower intraclass correlations (or higher DID magnitudes). When DID was ignored, the accuracy of the item discrimination estimates and standard errors was mainly problematic. Implications of the findings and limitations are discussed.

A More Flexible Bayesian Multilevel Bifactor Item Response Theory Model

“…Moreover, incorrect conclusions about the cluster effect could be made, such as underestimating the magnitude of the effect in the items and failing to detect the effect altogether when the overall effect is weak but the effect varies across the items such that the effect is strong in some items while weak in other items. These findings are consistent with those of other studies (Fujimoto, ; Lee & Cho, ; Zheng, ) and add to the minimal evidence that currently exists in this area, especially within a dual‐dependent context.…”

“…One reason is to obtain more accurate results. As research within a traditional multilevel IRT context (i.e., a non‐bifactor context) has shown, treating a varying cluster effect as constant leads to inaccurate estimates of the cluster effect and the item discriminations (Lee & Cho, ). Another reason for determining whether the cluster effect varies across items is because this issue has implications for the validity in the use and interpretation of scores (or dimensional estimates) as outlined in Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, ).…”

“…The cluster effect is assessed by calculating the intraclass correlation (ICC) for each item. An item's error‐free ICC specific to the primary dimension is calculated using (Lee & Cho, ; Muthén, ). This item‐level ICC is the proportion of the primary dimensional variance represented in item k that is attributable to the cluster level.…”

Cross-Level Invariance in Multilevel Factor Models

“…Previous simulation studies never explicitly focused on the effect of (not) applying cross-level invariance constraints in multilevel factor models. Some studies generated data with equal factor loadings, but did not apply cross-level constraints in the analysis (Kim, Yoon, Wen, Luo, & Kwok, 2015;Li & Beretvas, 2013), others generated data with different numbers of factors or different factor loadings across levels (Hox, Maas, & Brinkhuis, 2010;Lee & Cho, 2017), or focused on comparing frequentist and Bayesian estimation methods (Depaoli & Clifton, 2015;Guenole, 2016;HoltmanKoch, Lochner & Eid, 2016). The goal of the current article is to evaluate the effect of not applying cross-level invariance constraints in a multilevel latent mediation model on estimation problems in situations where the constraints are actually appropriate, in the frequentist framework.…”

The Bayesian Multilevel Trifactor Item Response Theory Model

“…When this is the case, the constraint in Equation 3 could result in inaccurate estimates for the item discriminations and misleading conclusions could be formed about the cluster effect represented in the data (Lee & Cho, 2017). A cluster effect that varies across the items has been more commonly discussed in the multilevel factor analytic literature (e.g., Mehta & Neale, 2005;Muthén, 1991Muthén, , 1994Rabe-Hesketh, Skrondal, & Pickles, 2004), though this discussion has recently appeared in unidimensional and multidimensional IRT contexts (Fujimoto, 2018;Lee & Cho, 2017). To allow the cluster effect to vary across the items in the proposed model, separate sets of item discriminations are estimated for Levels 2 and 3 (i.e., unconstraining the item discriminations across levels).…”