A Standardized Generalized Dimensionality Discrepancy Measure and a Standardized Model‐Based Covariance for Dimensionality Assessment for Multidimensional Models

Abstract

The standardized generalized dimensionality discrepancy measure and the standardized model-based covariance are introduced as tools to critique dimensionality assumptions in multidimensional item response models. These tools are grounded in a covariance theory perspective and associated connections between dimensionality and local independence. Relative to their precursors, they allow for dimensionality assessment in a more readily interpretable metric of correlations. A simulation study demonstrates the utility of the discrepancy measures' application at multiple levels of dimensionality analysis, and compares them to factor analytic and item response theoretic approaches. An example illustrates their use in practice.Approaches to dimensionality assessment based on item response theory (IRT) have typically been framed as either confirmatory approaches to evaluate unidimensionality or exploratory approaches that seek to determine the multidimensional structure. Such procedures are well suited to situations where assessment purposes and design imply the use of a unidimensional model. However, this strategy leaves something to be desired in situations where substantive theory, prior research, and assessment design dictate that performance depends on multiple aspects of proficiency and the analyst wishes to employ a multidimensional IRT (MIRT) model. This paper focuses on this situation, namely, where the analyst has specified the model, including the number of latent variables and the pattern of dependence of the item responses on those latent variables, a priori (e.g.Drawing from terminology in factor analysis, we refer to such situations as confirmatory MIRT modeling. Recent advances in procedures in fit for IRT broadly conceptualized (Maydeu-Olivares, 2013) have promise for applications in MIRT. The current work focuses on one such aspect of examining data-model fit, namely, in terms of the assumed dimensionality of the model.To make this more concrete, consider the multidimensional normal-ogive IRT model (e.g., McDonald, 1999) for dichotomous observables (i.e., scored item 144 A Standardized Generalized Dimensionality Discrepancy Measure responses) which specifies the probability of an observed value of 1 (corresponding to a particular response) as P(X i j = 1|θ i , a j , d j ) = F(a j θ i + d j ),(1)

Model Criticism of Growth Curve Models via Posterior Predictive Model Checking

“…To date, the SGDDM M has neither been used in applied research nor has it been systematically investigated in any methodological work, but it is included here as a method of to evaluate the marginal mean structure. In contrast, the SGDDM C has been used with success in applied psychometric research (e.g., Levy, Crawford, Fay, & Poole, 2011;Rupp et al, 2012) and investigated methodologically (e.g., Crawford, 2014;Levy et al, 2015) in the context of psychometric models. Although the SGGDM C has exhibited a strong performance for different types of highly complex models, it remains an open question how it will perform in the context of GCM.…”

“…dimensionality discrepancy measure (SGDDM;Levy, Xu, Yel, & Svetina, 2015; seeLevy & Svetina, 2011 for the unstandardized version, GDDM). The SGDDM C is an aggregation of conditional associations, specifically those elements in the off-diagonal of the R matrix.…”

Bayesian Dimensionality Assessment for the Multidimensional Nominal Response Model

“…One readily interpretable model evaluation statistic is the standardized generalized dimensionality discrepancy measure (SGDDM), introduced by Levy et al ( 2015 ) as a variant of the procedure in Levy et al ( 2009 ). The SGDDM is a quantification of the standardized model-based covariance between two items, j and j ′, and thus, it is interpretable as a model-based posterior correlation between a pair of responses.…”

“…We have used only a limited number of samples because MCMC is highly time consuming and the simulation study has to be kept within our limit of computational resources. The figure of 50 samples was taken from Levy et al ( 2015 ), who ran similar simulations and pointed out that this figure is sufficient to identify broad patterns in the data.…”

“…We have applied two model evaluation statistics that have been recently proposed in a Bayesian statistical context, the widely applicable information criterion (WAIC) and the leave-one-out cross-validation (LOO), based on the information theory (Gelman et al, 2014b ), and which, to our knowledge, have not been applied before in a psychometric context. Moreover, the article includes an extension to the nominal case of the standardized generalized dimensionality discrepancy measure (SGDDM) by Levy et al ( 2015 ). The SGDDM was originally proposed for the evaluation of the dichotomous item response model, and subsequently extended to ordinal factorial models.…”