Nonparametric CAT for CD in Educational Settings With Small Samples

Abstract

Cognitive diagnostic computerized adaptive testing (CD-CAT) has been suggested by researchers as a diagnostic tool for assessment and evaluation. Although model-based CD-CAT is relatively well researched in the context of large-scale assessment systems, this type of system has not received the same degree of research and development in small-scale settings, such as at the course-based level, where this system would be the most useful. The main obstacle is that the statistical estimation techniques that are successfully applied within the context of a large-scale assessment require large samples to guarantee reliable calibration of the item parameters and an accurate estimation of the examinees' proficiency class membership. Such samples are simply not obtainable in course-based settings. Therefore, the nonparametric item selection (NPS) method that does not require any parameter calibration, and thus, can be used in small educational programs is proposed in the study. The proposed nonparametric CD-CAT uses the nonparametric classification (NPC) method to estimate an examinee's attribute profile and based on the examinee's item responses, the item that can best discriminate the estimated attribute profile and the other attribute profiles is then selected. The simulation results show that the NPS method outperformed the compared parametric CD-CAT algorithms and the differences were substantial when the calibration samples were small.

Stratified Item Selection Methods in Cognitive Diagnosis Computerized Adaptive Testing

“…Within each subfigure, the item selection methods with novel stratification index performed slightly better than the item selection methods with original stratification index, and the item selection methods without stratification performed the worst, which indicated the efficiency of the novel stratification index proposed in this study. In addition, as found by Chang et al (2018), the S-WNPS and S-NPS methods performed similarly, and both outperformed the S-PWKL item selection method. Furthermore, the S-NPS and S-WNPS methods without stratification (i.e., NPS and WNPS) considerably outperformed the parametric method (i.e., PWKL), and their corresponding PARs stayed high.…”

“…Xu et al, 2016) and the item bank size J = 350. The test length was L = 4 K (Chang et al, 2018). The Q-matrix is a J × K matrix with q jk ~ i . i . d . Bernouli ( 0 . 5 ) for all j = 1, .…”

“…In addition, Chang et al (2018) proposed the nonparametric item selection (NPS) method and the weighted nonparametric item selection (WNPS) method to select “best-fitting” items for each examinee, which only required specification of Q-matrix and did not depend on any statistical model, for educational settings with small samples. And Chang et al (2018) illustrated that the NPS and WNPS methods outperformed the PWCDI and PWKL methods across all simulated conditions.…”

“…In addition, Chang et al (2018) proposed the nonparametric item selection (NPS) method and the weighted nonparametric item selection (WNPS) method to select “best-fitting” items for each examinee, which only required specification of Q-matrix and did not depend on any statistical model, for educational settings with small samples. And Chang et al (2018) illustrated that the NPS and WNPS methods outperformed the PWCDI and PWKL methods across all simulated conditions. It was worth noting that the nonparametric classification (NPC) method, developed by Chiu and Douglas (2013), was required in the NPS and WNPS methods to obtain the estimates of examinees’ attribute patterns in the CD-CAT process.…”

On the Design of Interim Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT) in Learning Context

“…In particular, they derived the minimum number of items required in order to identify the attribute pattern of a student as well as the specific types of initial items that are required to reach the optimal classification results, under both ideal and practical scenarios. Their proposal was later used by Chang, Chiu, and Tsai (2018) in their nonparametric CD-CAT design. While Xu et al (2016)'s results are suitable for independent attribute structures, we extend the results to scenarios with attribute hierarchies.…”