Bridging Parametric and Nonparametric Methods in Cognitive Diagnosis

Ma, Chenchen; Torre, Jimmy de la; Xu, Gongjun

doi:10.1007/s11336-022-09878-2

Cited by 8 publications

(17 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A disrupted estimation will lead to biased parameters, less precise attribute classifications, and overestimated classification accuracy (Kreitchmann et al, 2022;Ma & Jiang, 2021;Sen & Cohen, 2021). To address this, nonparametric CDM was proposed as a suitable alternative to provide accurate attribute profile classifications under those challenging conditions that disrupt parameter estimation (e.g., small sample size, low-quality items, complex Q-matrices; Chiu et al, 2018;Ma et al, 2022;Oka & Okada, 2021). However, up until today, it was not possible to derive the likelihood nor posterior probabilities from these methods, which prevented from calculating fit and reliability indices.…”

Section: Discussionmentioning

confidence: 99%

The Restricted DINA Model: A Comprehensive Cognitive Diagnostic Model for Classroom-Level Assessments

Nájera¹,

Abad

Chiu

et al. 2023

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

The nonparametric classification (NPC) method has been proven to be a suitable procedure for cognitive diagnostic assessments at a classroom level. However, its nonparametric nature impedes the obtention of a model likelihood, hindering the exploration of crucial psychometric aspects, such as model fit or reliability. Reporting the reliability and validity of scores is imperative in any applied context. The present study proposes the restricted deterministic input, noisy “and” gate (R-DINA) model, a parametric cognitive diagnosis model based on the NPC method that provides the same attribute profile classifications as the nonparametric method while allowing to derive a model likelihood and, subsequently, to compute fit and reliability indices. The suitability of the new proposal is examined by means of an exhaustive simulation study and a real data illustration. The results show that the R-DINA model properly recovers the posterior probabilities of attribute mastery, thus becoming a suitable alternative for comprehensive small-scale diagnostic assessments.

show abstract

Section: Discussionmentioning

confidence: 99%

The Restricted DINA Model: A Comprehensive Cognitive Diagnostic Model for Classroom-Level Assessments

Nájera¹,

Abad

Chiu

et al. 2023

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

show abstract

“…The initial sample consisted of 131 participants, but was then reduced to 119, since 11 of them did not complete the test. Given the small sample size, we used DINA (deterministic inputs, noisy “and” gate) and RRUM (reduced reparametrized unified model) under a nonparametric approximation for the specific DCM (Chiu & Douglas, 2013 ; Chiu et al, 2018 ; Ma et al, 2022 ; Sen & Cohen, 2021 ).…”

Section: Methodsmentioning

confidence: 99%

“…Selecting an appropriate cognitive diagnostic model is necessary for an accurate diagnosis or classification of participants (Tatsuoka, 2009 ). Thus, since all the items considered in this study correspond to a single attribute, only the non-compensatory models DINA and RRUM (De la Torre & Minchen, 2014 ) were used under a nonparametric approximation given the small sample size (Chiu & Douglas, 2013 ; Chiu et al, 2018 ; Ma et al, 2022 ).…”

Section: Fitting Modelmentioning

confidence: 99%

“…, where the latent response variable is related to the performance in items Y ij , considering the randomness parameter g j = P(Y ij = 1| ij = 0) and the slope parameter s j = P(Y ij = 0| ij = 1) . Thus, assuming independence locally and among the TC, the joint probability function for all the answers would be defined by: However, and under the parametric perspective, these models mainly use the marginal maximum likelihood estimation based on the maximum expectation algorithm (MMLE-EM) or the Marcok Chain Monte Carlo (MCMC) techniques, which work well in large-scale evaluations where thousands of subjects are available or take a long time to estimate with small samples (Ma et al, 2022). Faced with these conditions, Chiu and Douglas (2013) propose a nonparametric classification (NPC) model, which avoids parametric estimation by classifying subjects and minimizing the distance between the observed subject responses on an item and the centers of the latent classes, nominated ideal response vectors.…”

Section: Fitting Modelmentioning

confidence: 99%

See 1 more Smart Citation

Nonparametric cognitive diagnosis of profiles of mathematical knowledge of teacher education candidates

et al. 2023

View full text Add to dashboard Cite

Traditionally, the selection process of teacher candidates has emphasized the assessment of subject matter and pedagogical knowledge using psychometric methodologies, which simply organize candidates in continuous scales and require a large number of samples. However, these methods do not allow for the identification of candidates’ knowledge profiles and learning paths, which would help develop programs tailored to support students in their training process. In this study, an evaluation instrument was developed by using the nonparametric approach to model diagnostic classifications and was then validated on a sample of 119 participants. This instrument allows for disaggregating candidates’ initial knowledge and establishing relationships between its components. The results showed that candidates present a variety of profiles, which may consider more than one attribute. Not only does it provide a score that can be used for selection processes, it also provides useful information for initial teacher training methods.

show abstract

“…Specifically, current literature includes CDM applications not only in mathematics (e.g., Y.-H. Chen et al, 2019;Tang & Zhan, 2020), reading (e.g., George & Robitzsch, 2021), or foreign language evaluation (e.g., Dong et al, 2021;Du & Ma, 2021), but also for assessing personality (e.g., Huang, 2022;Revuelta et al, 2018), psychological disorders (e.g., de la Torre et al, 2018;Xi et al, 2020) or work and study attitudes (e.g., García et al, 2014;Sorrel et al, 2016). Additionally, CDMs are currently being implemented across heterogeneous conditions (Sessoms & Henson, 2018), with sample sizes as small as 44 (Jang et al, 2015) and up to 71,000 respondents (George & Robitzsch, 2014) , with recent simulation studies supporting the use of parametric CDM methods for sample sizes as small as 100 (e.g., Ma et al, 2022;Ma & Jiang, 2021). In fact, there is a growing trend towards implementation of CDMs with small samples (e.g., Fan et al, 2021;Tang & Zhan, 2021), as they constitute a common context for diagnostic assessment in which tailored feedback and remediation can be easily provided.…”

mentioning

confidence: 99%

Improving reliability estimation in cognitive diagnosis modeling

et al. 2022

Self Cite

View full text Add to dashboard Cite

Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability estimation is crucial for valid score interpretations. In this sense, most CDM reliability indices are based on the posterior probabilities of the estimated attribute profiles. These posteriors are traditionally computed using point estimates for the model parameters as approximations to their populational values. If the uncertainty around these parameters is unaccounted for, the posteriors may be overly peaked, deriving into overestimated reliabilities. This article presents a multiple imputation (MI) procedure to integrate out the model parameters in the estimation of the posterior distributions, thus correcting the reliability estimation. A simulation study was conducted to compare the MI procedure with the traditional reliability estimation. Five factors were manipulated: the attribute structure, the CDM model (DINA and G-DINA), test length, sample size, and item quality. Additionally, an illustration using the Examination for the Certificate of Proficiency in English data was analyzed. The effect of sample size was studied by sampling subsets of subjects from the complete data. In both studies, the traditional reliability estimation systematically provided overestimated reliabilities, whereas the MI procedure offered more accurate results. Accordingly, practitioners in small educational or clinical settings should be aware that the reliability estimation using model parameter point estimates may be positively biased. R codes for the MI procedure are made available

show abstract

Bridging Parametric and Nonparametric Methods in Cognitive Diagnosis

Cited by 8 publications

References 30 publications

The Restricted DINA Model: A Comprehensive Cognitive Diagnostic Model for Classroom-Level Assessments

The Restricted DINA Model: A Comprehensive Cognitive Diagnostic Model for Classroom-Level Assessments

Nonparametric cognitive diagnosis of profiles of mathematical knowledge of teacher education candidates

Improving reliability estimation in cognitive diagnosis modeling

Contact Info

Product

Resources

About