Observed score reliability indices in diagnostic classification models

Abstract

Quantifying the reliability of latent variable estimates in diagnostic classification models has been a difficult topic, complicated by the classification-based nature of these models. In this study, we derive observed score reliability indices based on diagnostic classification models as an extension of classical test theory-based reliability. Additionally, we derive conditional observed sum-and sub-score distributions. In this manner, various conditional expectations and conditional standard error of measurement estimates can be calculated for both sum-and sub-scores of a test. The proposed methods provide a variety of expectations and standard errors for attribute estimates, which we demonstrate with an analysis of an empirical test. Moreover, a simulation study revealed the proposed sub-score based reliability index was correlated to a previously developed attribute mastery reliability index.

Modeling Partial Knowledge in Multiple-Choice Cognitive Diagnostic Assessment

“…For example, the estimates for latent classes may not be reliable because each attribute has only a few items. In theory, it should be possible to calculate an observed reliability index in a same manner with Yamaguchi and Templin (2022). However, the actual derivation and computation of the reliability index for a specific multiple-choice CDM requires a substantial amount of effort that deserves a separate study, as in the case of the abovementioned paper.…”