Improving Robustness in Q-Matrix Validation Using an Iterative and Dynamic Procedure

Abstract

In the context of cognitive diagnosis models (CDMs), a Q-matrix reflects the correspondence between attributes and items. The Q-matrix construction process is typically subjective in nature, which may lead to misspecifications. All this can negatively affect the attribute classification accuracy. In response, several methods of empirical Q-matrix validation have been developed. The general discrimination index (GDI) method has some relevant advantages such as the possibility of being applied to several CDMs. However, the estimation of the GDI relies on the estimation of the latent group sizes and success probabilities, which is made with the original (possibly misspecified) Q-matrix. This can be a problem, especially in those situations in which there is a great uncertainty about the Q-matrix specification. To address this, the present study investigates the iterative application of the GDI method, where only one item is modified at each step of the iterative procedure, and the required cutoff is updated considering the new parameter estimates. A simulation study was conducted to test the performance of the new procedure. Results showed that the performance of the GDI method improved when the application was iterative at the item level and an appropriate cutoff point was used. This was most notable when the original Q-matrix misspecification rate was high, where the proposed procedure performed better 96.5% of the times. The results are illustrated using Tatsuoka’s fraction-subtraction data set.

Balancing fit and parsimony to improve Q‐matrix validation

“…In this regard, we would like to emphasize that Q‐matrix validation methods should not be blindly trusted. As indicated in the caveats and recommendations made by Nájera et al (2020), Q‐matrix validation methods should not be understood as a substitute for experts’ judgement, but a complement to it.…”

“…No re‐estimation of the CDM is made after introducing the modifications in the Q‐matrix. The GDI and Hull methods were implemented iteratively (Nájera, Sorrel, de la Torre, & Abad, 2020). This means that the CDM is re‐estimated after each modification.…”

Empirical study on the construction of a cognitive model of factorization in eighth-grade students

“…Students often made mistakes due to their incomprehension of the meaning of factorization; the third is the test Q matrix's impact on cognitive diagnosis results. Setting the correct Q matrix is the key factor for obtaining accurate parameter estimation results (Nájera et al, 2020), but these test attributes in the test project theory do not strictly follow the cognitive attributes' hierarchical structure in the cognitive model. The cognitive model reflects subjects' logic sequence, which they must obey in the process of knowledge mastery, but the project test mode does not necessarily follow this logic sequence.…”

Cognitively Diagnostic Analysis Using the G-DINA Model in R

“…To validate the number of attributes, Nájera and colleagues [27] adopted procedures for assessing the dimensionality, which were initially developed for exploratory analysis, often without a provisional Q-matrix. When the number of attributes has been validated, a host of methods have been developed for identifying misspecified elements [28][29][30][31]. De la Torre and Minchen [32] recommended employing a saturated CDM when conducting Q-matrix validation to avoid conflating Q-matrix misspecifications with model misspecifications.…”

“…Alternatively, the PVAF (i.e., the proportion of variance accounted for) method with fixed or predicted cutoffs can be applied [28] when using this function. In the cdmTools package, Q-matrix validation can be performed with cdmTools::valQ() function, which implements the Hull method with PVAF or McFadden's pseudo R-squared [47] with various iteration algorithms [31].…”