How Adaptive Is an Adaptive Test: Are All Adaptive Tests Adaptive?

Abstract

The primary distinguishing feature of computerized adaptive testing (CAT) is that sets of items that are administered to examinees are specifically selected for each examinee during the test process. This implies that examinees who respond differently to items on a test will get different sets of test items. However, if an item bank is small, if there are strict content constraints, or if there is strong exposure control, there might not be much adaptation during the test. Examinees who perform differently might get many of the same test items. The research reported here recommends some statistical indicators of how much adaptation is taking place and shows how these indicators vary for different kinds of adaptive test designs. Guidelines are provided for the values of the statistics indicating that a CAT is strongly adaptive.

Three Measures of Test Adaptation Based on Optimal Test Information

“…There are several existing approaches to quantifying the adaptive nature of a CAT. Reckase, Ju, and Kim (2019) proposed three adaptation indices based on the variance of the difficulty (b) parameters of administered items:…”

“…This is consistent with the reason for their development, which was to show the difference in adaptivity for different test designs. Taken together, the three indices developed by Reckase and his co-authors do an excellent job of this, as "they provide objective information about the matching of the item selection to the level of performance for everyone taking the test" (Reckase et al, 2019).…”

“…For example, if the optimal item had a = 1.0 and c = 0.20, it would need a b value that was 0.27 θ units below the individual's true θ level to provide the maximum information (where the offset from the true θ level would vary with optimal values chosen for a and c). This could be done using Equation 2 from Reckase et al (2019). This type of approach would create an "optimal" value for the item parameters for the theoretical items, which in turn would create the information values needed to calculate the EOI and TOI indices.…”

A Framework for Measuring the Amount of Adaptation of Rasch‐based Computerized Adaptive Tests

“…However, an important question is what happens when the item bank does not contain enough items with an appropriate range of content and statistical properties to satisfy the CAT test specifications. Kim, Ju, and Reckase (2018) and Reckase, Ju, and Kim (2018, 2019) pointed out that in these cases the CAT may begin to look more like a fixed‐length test and may not exhibit the desired level of adaptation. They defined a test as exhibiting a high amount of adaptation when the items selected during the CAT well match the final ability estimate of the examinee.…”

“…Somewhat different than Reckase et al. (2018, 2019), Kingsbury and Wise's (2018) index could be applied at the individual level. While the indices suggested by Kim et al.…”