Robustness of Adaptive Measurement of Change to Item Parameter Estimation Error

Abstract

Adaptive measurement of change (AMC) is a psychometric method for measuring intra-individual change on one or more latent traits across testing occasions. Three hypothesis tests—a Z test, likelihood ratio test, and score ratio index—have demonstrated desirable statistical properties in this context, including low false positive rates and high true positive rates. However, the extant AMC research has assumed that the item parameter values in the simulated item banks were devoid of estimation error. This assumption is unrealistic for applied testing settings, where item parameters are estimated from a calibration sample before test administration. Using Monte Carlo simulation, this study evaluated the robustness of the common AMC hypothesis tests to the presence of item parameter estimation error when measuring omnibus change across four testing occasions. Results indicated that item parameter estimation error had at most a small effect on false positive rates and latent trait change recovery, and these effects were largely explained by the computerized adaptive testing item bank information functions. Differences in AMC performance as a function of item parameter estimation error and choice of hypothesis test were generally limited to simulees with particularly low or high latent trait values, where the item bank provided relatively lower information. These simulations highlight how AMC can accurately measure intra-individual change in the presence of item parameter estimation error when paired with an informative item bank. Limitations and future directions for AMC research are discussed.

Untitled

“…Each simulation condition was replicated 1,000 times to represent 1,000 simulees for each combination of pattern, 𝜃 1 value, and CAT item bank. In addition, true (error-free) item parameters were used in the simulations because Cooperman et al (2021) showed that item parameter estimation error had a small to negligible effect on AMC performance in similar testing scenarios.…”

“…Their common practice was to specify a limited number of change patterns that represented different arbitrary combinations of magnitude, direction, and linear or nonlinear patterns of change. However, Cooperman et al (2021) showed that among design factors including hypothesis testing method, item parameter estimation error, starting 𝜃 value, and change pattern, the effect sizes of change pattern on most AMC performance indicators were far greater than other design factors. For example, in their results, the classical effect sizes ( 2 ) for true positive rates (TPRs, or power) were 0.793 for change pattern, 0.06 for starting 𝜃, and 0.049 for the change pattern × starting 𝜃 interaction.…”

“…They concluded that AMC outperformed all three CTT methods in the examined conditions. Not only did AMC & Thomas, 1991), and two new hypothesis testing methods-a Z-test and a likelihood-ratio chi-square test. They found improvements in the procedure by showing better adherence to Type I error rates and better power for detecting small changes.…”

“…Wang et al (2021) further developed and investigated the performance of AMC methods for multidimensional latent traits measured across multiple occasions, which is the most comprehensive AMC framework. Cooperman et al (2021) showed that AMC is generally robust to item parameter estimation error.…”

“…For example, Finkelman et al (2010) showed that in the case of two testing occasions, TPRs could be as low as around 0.4 when the magnitude of change was 0.5 and as high as nearly 1.0 when the magnitude of change was 1.5. Cooperman et al (2021) showed that in the case of four testing occasions, change patterns [+0. 25, +0.25, +0.25], [+0.5, +0.25, 0], and [+0.75, 0, 0] resulted in TPRs of around 0.55, 0.60, and 0.65, respectively; in other words, patterns with the same total magnitude of change but different distributive patterns resulted in different TPRs.…”