Adaptive Weight Estimation of Latent Ability: Application to Computerized Adaptive Testing With Response Revision

Abstract

An adaptive weight estimation approach is proposed to provide robust latent ability estimation in computerized adaptive testing (CAT) with response revision. This approach assigns different weights to each distinct response to the same item when response revision is allowed in CAT. Two types of weight estimation procedures, nonfunctional and functional weight, are proposed to determine the weight adaptively based on the compatibility of each revised response with the assumed statistical model in relation to remaining observations. The application of this estimation approach to a data set collected from a large-scale multistage adaptive testing demonstrates the capability of this method to reveal more information regarding the test taker’s latent ability by using the valid response path compared with only using the very last response. Limited simulation studies were concluded to evaluate the proposed ability estimation method and to compare it with several other estimation procedures in literature. Results indicate that the proposed ability estimation approach is able to provide robust estimation results in two test-taking scenarios.

Exploration of Latent Structure in Test Revision and Review Log Data

“…Furthermore, scoring based on the number of attempts instead of first‐attempt correctness also improves measurement reliability for target ability (Attali & Powers, 2010; Bergner et al., 2019). Recent studies on computerized adaptive testing (CAT) allowing for response review also suggested that more information regarding examinees' proficiency and test‐taking behavior can be gained compared with CAT without such a design feature (Wang et al., 2021). On the other hand, allowing response revision and review remarkably complexifies the observed data structure.…”

“…When a CBT allows reviews, revision behavior can often explicitly or implicitly influence a test taker's or a subgroup's performance. In some measurement models, answer change trajectory is explicitly used for scoring (e.g., Attali & Powers, 2010; Wang et al., 2021). Revision behaviors may also implicitly affect an individual's score.…”