Three Measures of Test Adaptation Based on Optimal Test Information

Abstract

This study extends the work of Reckase, Zu, and Kim (2019) by introducing three new measures of test adaptation. These measures are based on the amount of information that the test provides to individual examinees and are designed to provide theoretical, engineering, and operational views of the amount of adaptation that a particular test provides. The three measures are applied to simulations of adaptive, multistage, and fixedform tests to describe how the measures could be used in an operational setting to identify difficulties with adaptation and to suggest processes for improving adaptivity.

A Framework for Measuring the Amount of Adaptation of Rasch‐based Computerized Adaptive Tests

“…Kingsbury and Wise (2018) offered a fourth index related to Reckase et al.’s framework that is based on taking the ratio of the actual test information from the selected items over the optimal test information that could be observed given the item pool. The POI index can be calculated as: …”

“…They applied their indices to evaluate adaptation in CATs that used the Rasch (1960) and 3PL (Birnbaum, 1968) models. Kingsbury and Wise (2018) suggested a different index for measuring CAT adaptation based on the same framework that involved taking the ratio of the test information function for the selected items over the optimal test information function that could be obtained during the CAT. Somewhat different than Reckase et al.…”

“…Somewhat different than Reckase et al. (2018, 2019), Kingsbury and Wise's (2018) index could be applied at the individual level. While the indices suggested by Kim et al.…”

“…(2018), Reckase et al. (2018, 2019), and Kingsbury and Wise (2018) offer useful statistics for measuring the amount of adaptation of a CAT, they are not the only options to measure adaptation.…”