Adapting cognitive diagnosis computerized adaptive testing item selection rules to traditional item response theory

Abstract

Currently, there are two predominant approaches in adaptive testing. One, referred to as cognitive diagnosis computerized adaptive testing (CD-CAT), is based on cognitive diagnosis models, and the other, the traditional CAT, is based on item response theory. The present study evaluates the performance of two item selection rules (ISRs) originally developed in the CD-CAT framework, the double Kullback-Leibler information (DKL) and the generalized deterministic inputs, noisy "and" gate model discrimination index (GDI), in the context of traditional CAT. The accuracy and test security associated with these two ISRs are compared to those of the point Fisher information and weighted KL using a simulation study. The impact of the trait level estimation method is also investigated. The results show that the new ISRs, particularly DKL, could be used to improve the accuracy of CAT. Better accuracy for DKL is achieved at the expense of higher item overlap rate. Differences among the item selection rules become smaller as the test gets longer. The two CD-CAT ISRs select different types of items: items with the highest possible a parameter with DKL, and items with the lowest possible c parameter with GDI. Regarding the trait level estimator, expected a posteriori method is generally better in the first stages of the CAT, and converges with the maximum likelihood method when a medium to large number of items are involved. The use of DKL can be recommended in low-stakes settings where test security is less of a concern.

Improving Accuracy and Usage by Correctly Selecting: The Effects of Model Selection in Cognitive Diagnosis Computerized Adaptive Testing

“…The lack of appropriate software programs for conducting adaptive applications based on CDM might have hampered the development of more empirical applications. To facilitate these applications and encourage new ones, the code was turned into an R package named (Sorrel, Nájera, & Abad, 2020).…”

“…Model estimation was conducted using the R package (Ma & de la Torre, 2017). Finally, we developed the R package (Sorrel, Nájera, & Abad, 2020) that allows for application of the 2LR and CD-CAT analyses in R .…”

“…Instead, the index considers all the information available at the current step (i.e., π ( α lj * ) ( t ) ). This is one strength of the method as it allows avoiding the problem of multiple maxima that is likely to occur at the initial steps of a CAT (Sorrel, Barrada, et al, 2020).…”

A new approach to assessing shyness of college students using computerized adaptive testing: CAT-Shyness

“…Furthermore, we chose the maximum Fisher information item selection rule because of its popularity and availability (Magis & Barrada, 2017). However, this rule has a slightly worse performance in terms of accuracy, compared to global information item selection rules (Sorrel et al, 2020), and differences are greater for extreme latent trait levels (i.e., <−1.5, >1.5). Future studies could use different item selection rules, such as the global information item selection rule, to improve test accuracy; and the CAT could be probably shorter with a different item selection rule.…”

“…The simplicity and stability of the EAP makes it a widely used method for CAT simulations (e.g., Bulut & Kan, 2012; Chen, Hou, & Dodd, 1998). Third, the accuracy of EAP estimates are higher than the MLE (e.g., Sorrel, Barrada, de la Torre, & Abad, 2020).…”