Using a Projection IRT Method for Vertical Scaling When Construct Shift Is Present

Abstract

In vertical scaling, results of tests from several different grade levels are placed on a common scale. Most vertical scaling methodologies rely heavily on the assumption that the construct being measured is unidimensional. In many testing situations, however, such an assumption could be problematic. For instance, the construct measured at one grade level may differ from that measured in another grade (e.g., construct shift). On the other hand, dimensions that involve low‐level skills are usually mastered by almost all students as they progress to higher grades. These types of changes in the multidimensional structure, within and across grades, create challenges for developing a vertical scale. In this article, we propose the use of projective IRT (PIRT) as a potential solution to the problem. Assuming that a test measures a primary dimension of substantive interest as well as some peripheral dimensions, the idea underlying PIRT is to integrate out the secondary dimensions such that the model provides both item parameters and ability estimates for the primary dimension. A simulation study was conducted to evaluate the effectiveness of the PIRT as a method for vertical scaling. An example using empirical data from a measure of foundational reading skills is also presented.

Evaluation of the Linear Composite Conjecture for Unidimensional IRT Scale for Multidimensional Responses

“…Using PISA data as an example, the authors described a situation under which disproportionate cognitive demand across the dimensions is required for items that reside at certain regions (e.g., high ends) of the latent abilities. Another example is vertical scaling (Carlson, 2017; Strachan et al, in press). It is therefore possible that in some cases a nonlinear approximation of the composite will better represent the fitted UIRT.…”