Time-Efficient Adaptive Measurement of Change

Abstract

The adaptive measurement of change (AMC) refers to the use of computerized adaptive testing (CAT) at multiple occasions to efficiently assess a respondent's improvement, decline, or sameness from occasion to occasion. Whereas previous AMC research focused on administering the most informative item to a respondent at each stage of testing, the current research proposes the use of Fisher information per time unit as an item selection procedure for AMC. The latter procedure incorporates not only the amount of information provided by a given item but also the expected amount of time required to complete it. In a simulation study, the use of Fisher information per time unit item selection resulted in a lower false positive rate in the majority of conditions studied, and a higher true positive rate in all conditions studied, compared to item selection via Fisher information without accounting for the expected time taken. Future directions of research are suggested.

Statistical Applications in Educational Measurement

“…Items that are too time consuming are less preferred even if they are informative. Research has demonstrated that given a time limit, using MIT for item selection can yield more efficient CAT, leading to smaller measurement errors (Fan et al 2012, Finkelman & Wang 2019).…”

Using Response Time in Multidimensional Computerized Adaptive Testing

“…To broaden the applicability of methods focused on improving CAT delivery in terms of DTE, Cheng et al (2017) further modified the MIT criterion to a simple version (MIT-S) by using the expected logarithm of response time. Recently, the MIT framework was applied to adaptively measure individual change (Finkelman & Wang, 2019) and to calibrate items online (He et al, 2021), and was extended to the context of computerized classification testing (Sie et al, 2015) and cognitive diagnostic computerized adaptive testing (Huang, 2020). The results showed that items selected in the MIT framework saved testing time, ensured satisfactory measurement precision, and produced a higher true positive rate for measuring individual change.…”