A new Q‐matrix validation method based on signal detection theory

Abstract

The Q‐matrix is a crucial component of cognitive diagnostic theory and an important basis for the research and practical application of cognitive diagnosis. In practice, the Q‐matrix is typically developed by domain experts and may contain some misspecifications, so it needs to be refined using Q‐matrix validation methods. Based on signal detection theory, this paper puts forward a new Q‐matrix validation method (i.e., method) and then conducts a simulation study to compare the new method with existing methods. The results show that when the model is DINA (deterministic inputs, noisy ‘and’ gate), the method outperforms the existing methods under all conditions; under the generalized DINA (G‐DINA) model, the method still has the highest validation rate when the sample size is small, and the item quality is high or the rate of Q‐matrix misspecification is ≥.4. Finally, a sub‐dataset of the PISA 2000 reading assessment is analysed to evaluate the reliability of the method.