Sample Size Requirements for Applying Diagnostic Classification Models

Abstract

Results of a comprehensive simulation study are reported investigating the effects of sample size, test length, number of attributes and base rate of mastery on item parameter recovery and classification accuracy of four DCMs (i.e., C-RUM, DINA, DINO, and LCDMREDUCED). Effects were evaluated using bias and RMSE computed between true (i.e., generating) parameters and estimated parameters. Effects of simulated factors on attribute assignment were also evaluated using the percentage of classification accuracy. More precise estimates of item parameters were obtained with larger sample size and longer test length. Recovery of item parameters decreased as the number of attributes increased from three to five but base rate of mastery had a varying effect on the item recovery. Item parameter and classification accuracy were higher for DINA and DINO models.

The Restricted DINA Model: A Comprehensive Cognitive Diagnostic Model for Classroom-Level Assessments

“…Parametric CDM requires large sample sizes in order to obtain accurate item parameter estimates. A disrupted estimation will lead to biased parameters, less precise attribute classifications, and overestimated classification accuracy (Kreitchmann et al, 2022;Ma & Jiang, 2021;Sen & Cohen, 2021). To address this, nonparametric CDM was proposed as a suitable alternative to provide accurate attribute profile classifications under those challenging conditions that disrupt parameter estimation (e.g., small sample size, low-quality items, complex Q-matrices; Chiu et al, 2018;Ma et al, 2022;Oka & Okada, 2021).…”

Evaluation of the Computer-Based Orthographic Processing Assessment: An Application of Cognitive Diagnostic Modeling

“…The small and relatively homogeneous sample included in the present study reduces the confidence in our findings and certainly limits their generalizability. In partial defense of the study’s design, we employed a statistical approach that addressed our research questions and that has been shown reliable with small samples (Paulsen & Valdivia, 2021; Sen & Cohen, 2021). Nonetheless, future studies concerned with learning trajectories of orthographic processing skills would benefit from larger samples of children who vary more in their literacy abilities.…”

Nonparametric cognitive diagnosis of profiles of mathematical knowledge of teacher education candidates

“…The initial sample consisted of 131 participants, but was then reduced to 119, since 11 of them did not complete the test. Given the small sample size, we used DINA (deterministic inputs, noisy “and” gate) and RRUM (reduced reparametrized unified model) under a nonparametric approximation for the specific DCM (Chiu & Douglas, 2013 ; Chiu et al, 2018 ; Ma et al, 2022 ; Sen & Cohen, 2021 ).…”

“…Even though this study used non-parametrical models and the DINA rule to estimate parameters, given their success in simulations and empirical applications done with small samples (Chang et al, 2019 ; Paulsen & Valdivia, 2022 ), this method may still be considered a limitation, since it restrains the choice of models to be applied (Sen & Cohen, 2021 ). The same limitation is presented by the fact that the items only measured one attribute.…”