cluster analysis, cognitive diagnosis, latent class analysis,

In contrast to unidimensional item response models that postulate a single underlying proficiency, cognitive diagnosis models (CDMs) posit multiple, discrete skills or attributes, thus allowing CDMs to provide a finer-grained assessment of examinees' test performance. A common component of CDMs for specifying the attributes required for each item is the Q-matrix. Although construction of Q-matrix is typically performed by domain experts, it nonetheless, to a large extent, remains a subjective process, and misspecifications in the Q-matrix, if left unchecked, can have important practical implications. To address this concern, this paper proposes a discrimination index that can be used with a wide class of CDM subsumed by the generalized deterministic input, noisy "and" gate model to empirically validate the Q-matrix specifications by identifying and replacing misspecified entries in the Q-matrix. The rationale for using the index as the basis for a proposed validation method is provided in the form of mathematical proofs to several relevant lemmas and a theorem. The feasibility of the proposed method was examined using simulated data generated under various conditions. The proposed method is illustrated using fraction subtraction data.

Most methods for fitting cognitive diagnosis models to educational test data and assigning examinees to proficiency classes require the Q-matrix that associates each item in a test with the cognitive skills (attributes) needed to answer it correctly. In most cases, the Q-matrix is not known but is constructed from the (fallible) judgments of experts in the educational domain. It is widely recognized that a misspecification of the Q-matrix can negatively affect the estimation of the model parameters, which may then result in the misclassification of examinees. This article develops a Q-matrix refinement method based on the nonparametric classification method (Chiu & Douglas, in press), and comparisons of the residual sum of squares computed from the observed and the ideal item responses. The method is evaluated with three simulation studies and an application to real data. Results show that the method can identify and correct misspecified entries in the Q-matrix, thereby improving its accuracy.Keywords classification, cognitive diagnosis, Q-matrix, residual sum of squares, validityCognitive diagnosis models of educational test performance decompose overall ability in the domain of the test into a set of specific skills, called ''attributes,'' that an examinee may or may not possess, thereby providing a detailed description, or ''attribute pattern,'' of his or her strengths and weaknesses in the domain. The entire set of possible attribute patterns for a given test defines proficiency classes to which examinees can be assigned. Model-based methods use maximum likelihood estimation (MLE) procedures to estimate model parameters that are then used to assign examinees to proficiency classes (de la Torre

No abstract

The focus of cognitive diagnosis (CD) is on evaluating an examinee's strengths and weaknesses in terms of cognitive skills learned and skills that need study. Current methods for fitting CD models (CDMs) work well for large-scale assessments, where the data of hundreds or thousands of examinees are available. However, the development of CD-based assessment tools that can be used in small-scale test settings, say, for monitoring the instruction and learning process at the classroom level has not kept up with the rapid pace at which research and development proceeded for large-scale assessments. The main reason is that the sample sizes of the small-scale test settings are simply too small to guarantee the reliable estimation of item parameters and examinees' proficiency class membership. In this article, a general nonparametric classification (GNPC) method that allows for assigning examinees to the correct proficiency classes with a high rate when sample sizes are at the classroom level is proposed as an extension of the nonparametric classification (NPC) method (Chiu and Douglas in J Classif 30:225-250, 2013). The proposed method remedies the shortcomings of the NPC method and can accommodate any CDM. The theoretical justification and the empirical studies are presented based on the saturated general CDMs, supporting the legitimacy of using the GNPC method with any CDM. The results from the simulation studies and real data analysis show that the GNPC method outperforms the general CDMs when samples are small.