Relating Unidimensional IRT Parameters to a Multidimensional Response Space: A Review of Two Alternative Projection IRT Models for Scoring Subscales

Abstract

A practical concern for many existing tests is that subscore test lengths are too short to provide reliable and meaningful measurement. A possible method of improving the subscale reliability and validity would be to make use of collateral information provided by items from other subscales of the same test. To this end, the purpose of this article is to compare two different formulations of an alternative Item Response Theory (IRT) model developed to parameterize unidimensional projections of multidimensional test items: Analytical and Empirical formulations. Two real data applications are provided to illustrate how the projection IRT model can be used in practice, as well as to further examine how ability estimates from the projection IRT model compare to external examinee measures. The results suggest that collateral information extracted by a projection IRT model can be used to improve reliability and validity of subscale scores, which in turn can be used to provide diagnostic information about strength and weaknesses of examinees helping stakeholders to link instruction or curriculum to assessment results.

Unidimensional Interpretations for Multidimensional Test Items

“…The present results show that trait correlations higher than .70 might be sufficient to blur the threshold between unidimensionality and multidimensionality. This is consistent with the studies in the literature that suggest that unidimensional IRT model applications should be robust to violations of the unidimensionality assumption when test traits are highly correlated, that is, at or above .8 (Kahraman & Kamata, ; Kahraman & Thompson, ). Projection IRT model applications are not recommended when (1) test items do not show a complex factorial structure or (2) an observed complex factorial structure cannot be linked to test constructs defined by test blueprints.…”

“…This required approximations of true unidimensional item discrimination parameters of the complex‐structure items to be derived from the true two‐dimensional item discrimination parameters using the linear projection methodology (Wang, , ): where a i is a vector of item discrimination parameters of item i on dimensions k , Σ is a covariance matrix of θ with diagonal values of 1, and α is the reference composite, which is the first eigenvector of the corresponding item discrimination matrix with non‐negative α j such that α t Σ α = 1. Interested readers are referred to the original work of Wang (, 1988) and Zhang and Wang () for derivations of these equations and to Reckase () and Kahraman and Thompson () for further discussion of the methodology.…”

“…The Projection UIRT models allow individual (unidimensional) calibrations of test items with respect to a pre‐defined test dimension. The methodology originally was developed to extract construct‐relevant collateral information from items in relevant subscales to improve the reliability of target subscale scores (Ackerman & Davey, ; Davey & Hirsch, ; Kahraman & Kamata, ; Kahraman & Thompson, ). The current article, however, differs from the studies in the literature because it evaluates the usefulness of the Projection UIRT methodology in enhancing “intended” unidimensional interpretations of test performance.…”

“…where a i is a vector of item discrimination parameters of item i on dimensions k, is a covariance matrix of θ with diagonal values of 1, and α is the reference composite, which is the first eigenvector of the corresponding item discrimination matrix with non-negative α j such that α t α = 1. Interested readers are referred to the original work of Wang (1987Wang ( , 1988 and Zhang and Wang (1998) for derivations of these equations and to Reckase (1997) and Kahraman and Thompson (2011) for further discussion of the methodology. Figure 4 illustrates how true item discrimination parameters of the complexstructure items measuring an equally weighted test composite (45 • , a = [1, 1] reflecting this item's discrimination with respect to the first and the second test dimensions, respectively) diminishes (a* = .71) when projected onto the primary trait test composite.…”

Functionally Unidimensional Item Response Models for Multivariate Binary Data

“…More recently, de la Torre (2009) discussed the use of hierarchical unidimensional IRT models for domain-based tests. Kahraman and Thompson (2011) suggested two-stage methods for handling multidimensional data in which the first stage used pure unidimensional items and the second stage individually calibrated suspected multidimensional items using the scale established in the first stage as one of the dimensions. These strategies are all useful for different measurement circumstances.…”