Improving Accuracy and Usage by Correctly Selecting: The Effects of Model Selection in Cognitive Diagnosis Computerized Adaptive Testing

Abstract

Decisions on how to calibrate an item bank might have major implications in the subsequent performance of the adaptive algorithms. One of these decisions is model selection, which can become problematic in the context of cognitive diagnosis computerized adaptive testing, given the wide range of models available. This article aims to determine whether model selection indices can be used to improve the performance of adaptive tests. Three factors were considered in a simulation study, that is, calibration sample size, Q-matrix complexity, and item bank length. Results based on the true item parameters, and general and single reduced model estimates were compared to those of the combination of appropriate models. The results indicate that fitting a single reduced model or a general model will not generally provide optimal results. Results based on the combination of models selected by the fit index were always closer to those obtained with the true item parameters. The implications for practical settings include an improvement in terms of classification accuracy and, consequently, testing time, and a more balanced use of the item bank. An R package was developed, named cdcatR, to facilitate adaptive applications in this context.

On Bank Assembly and Block Selection in Multidimensional Forced-Choice Adaptive Assessments

“…As it is often done in CAT studies (e.g., Mulder & van der Linden, 2009), the item bank is assumed to be precisely calibrated, thus the true model parameters were taken as known to compute the trait scores. This allows to compute the upper-limit performance of the adaptive assessment and what is expected to obtain in practical settings provided the item parameters are accurately estimated (e.g., Sorrel et al, 2021). The R codes used for data generation and the simulation study are available from the corresponding author upon request.…”

Improving reliability estimation in cognitive diagnosis modeling

“…In this matrix, referred to as Q-matrix (Tatsuoka, 1983), each q-entry (q jk ) will receive a value of 1 or 0 depending on whether item j measures attribute k or not, respectively. The Q-matrix construction process is usually supervised by domain experts (e.g., Li & Suen, 2013;Sorrel et al, 2016), although several empirical Q-matrix estimation and validation methods have been proposed in the last years with the aim of reducing the degree of subjectivity involved in the task (e.g., de la Torre & Chiu, 2016;Nájera, Sorrel et al, 2021). The correct specification of the Q-matrix is of major importance since the presence of misspecifications can greatly disrupt the accuracy of attribute profile classifications (Gao et al, 2017;Rupp & Templin, 2008).…”

cdcatR: An R Package for Cognitive Diagnostic Computerized Adaptive Testing

“…Under challenging estimation conditions characterized, for example, by a small sample size, a larger estimation error can be expected to affect the performance of the CD-CAT [19]. To deal with this, Sorrel et al (2021) recommended performing a model comparison analysis at the item level in order to minimize the number of parameters to be estimated [20]. To this end, the authors used the two-step LR test, an efficient approach to the likelihood ratio test [21].…”

“…This allowed for a more direct comparison of the different CAT implementations. In applied contexts, some sampling error is to be expected (for a discussion of this, see, for example, [19,20]). The plots were obtained using the cdcat.summary() function of the package, which uses the R package ggplot2 [41].…”