Attribute‐Level and Pattern‐Level Classification Consistency and Accuracy Indices for Cognitive Diagnostic Assessment

Wang, Wenyi; Song, Lihong; Chen, Ping; Yu, Meng; Ding, Shuliang

doi:10.1111/jedm.12096

Cited by 51 publications

(98 citation statements)

References 35 publications

(70 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to absolute fit statistics, there is not a definite criterion for Pɑ and Pc values. In the light of the results obtained by Cui, Gierl, and Chang (2012), Wang, Song, Chen, Meng, and Ding (2015), and Johnson and Sinharay (2018), the values of accuracy and consistency are fairly high and acceptable in the current study. These moderate values of Pɑ and Pc at the wholepattern can be due to the presence of moderate correlations among the attributes (Cui et al, 2012).…”

Section: Model Fitsupporting

confidence: 68%

Diagnosing EFL learners’ writing ability: a diagnostic classification modeling analysis

2019

View full text Add to dashboard Cite

Cognitive diagnostic models (CDMs) have recently received a surge of interest in the field of second language assessment due to their promise for providing fine-grained information about strengths and weaknesses of test takers. For the same reason, the present study used the additive CDM (ACDM) as a compensatory and additive model to diagnose Iranian English as a foreign language (EFL) university students' L2 writing ability. To this end, the performance of 500 university students on a writing task was marked by four EFL teachers using the Empirically derived Descriptor-based Diagnostic (EDD) checklist. Teachers, as content experts, also specified the relationships among the checklist items and five writing sub-skills. The initial Q-matrix was empirically refined and validated by the GDINA package. Then, the resultant ratings were analyzed by the ACDM in the CDM package. The estimation of the skill profiles of the test takers showed that vocabulary use and content fulfillment are the most difficult attributes for the students. Finally, the study found that the skills diagnosis approach can provide informative and valid information about the learning status of students.

show abstract

Section: Model Fitsupporting

confidence: 68%

Diagnosing EFL learners’ writing ability: a diagnostic classification modeling analysis

2019

View full text Add to dashboard Cite

show abstract

“…In the model-based DIF checking, the Wald test ( Hou et al, 2014 ) was used. In the testing of reliability and validity, the classification accuracy ( P a ) and consistency ( P c ) indices ( Wang et al, 2015 ) were computed.…”

Section: Development Of the Instrument For Longitudinal Learning Diagmentioning

confidence: 99%

The Development of an Instrument for Longitudinal Learning Diagnosis of Rational Number Operations Based on Parallel Tests

Tang

Zhan

2020

Front. Psychol.

View full text Add to dashboard Cite

The precondition of the measurement of longitudinal learning is a high-quality instrument for longitudinal learning diagnosis. This study developed an instrument for longitudinal learning diagnosis of rational number operations. In order to provide a reference for practitioners to develop the instrument for longitudinal learning diagnosis, the development process was presented step by step. The development process contains three main phases, the Q-matrix construction and item development, the preliminary/pilot test for item quality monitoring, and the formal test for test quality control. The results of this study indicate that (a) both the overall quality of the tests and the quality of each item are good enough and that (b) the three tests meet the requirements of parallel tests, which can be used as an instrument for longitudinal learning diagnosis to track students’ learning.

show abstract

“…However, the traditional reliability and validity concepts introduced in classical test theory and item response theory have not yet been fully applied to cognitive diagnostic assessment, which mainly focuses on making classification decisions about discrete latent attributes (Henson & Douglas, ; Henson, Roussos, Douglas, & He, ). Therefore, research on reliability and validity in the context of cognitive diagnostic assessment is a relatively new topic, which has drawn increasing attention in the field of educational measurement recently (Cui, Gierl, & Chang, ; Ding, Mao, Wang, Luo, & Cui, ; Gierl, Cui, & Zhou, ; Leighton & Gierl, ; Rupp, Templin, & Henson, ; Tatsuoka, ; Templin & Bradshaw, ; Wang, Song, Chen, Meng, & Ding, ).…”

mentioning

confidence: 99%

“…Existing literature on this topic, including books and articles, has established several indices for evaluating the reliability and validity in cognitive diagnostic assessment (e.g., Cui et al., ; Ding et al., ; Gierl et al., ; Rupp et al., ; Tatsuoka, ; Templin & Bradshaw, ; Wang et al., ). Most of them are post hoc indices, because they can only be calculated at the end of the tests when all test‐takers’ response data are collected.…”

mentioning

confidence: 99%

“…Because the theoretical construct validity index is based on the ideal or expected examinee response patterns (Leighton, Gierl, & Hunka, ) to evaluate the quality of the Q ‐matrix design for a diagnostic test, it does not take the item parameters (e.g., slip and guessing parameters) into account when evaluating the reliability or validity of a test. However, it is necessary to consider the information from the item parameters because they have been shown to affect reliability and validity (Cui et al., ; Gierl et al., ; Wang et al., ). Other indices (i.e., CDI, ADI, MCDI, and MADI), which use Kullback‐Leibler information (Chang & Ying, ) as a measure of an item's (or test's) discrimination power, are effective for test construction.…”

mentioning

confidence: 99%

See 1 more Smart Citation

An Item‐Level Expected Classification Accuracy and Its Applications in Cognitive Diagnostic Assessment

Wang

Song

Chen

et al. 2019

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

Most of the existing classification accuracy indices of attribute patterns lose effectiveness when the response data is absent in diagnostic testing. To handle this issue, this article proposes new indices to predict the correct classification rate of a diagnostic test before administering the test under the deterministic noise input “and” gate (DINA) model. The new indices include an item‐level expected classification accuracy (ECA) for attributes and a test‐level ECA for attributes and attribute patterns, and both of them are calculated based solely on the known item parameters and Q‐matrix. Theoretical analysis showed that the item‐level ECA could be regarded as a measure of correct classification rates of attributes contributed by an item. This article also illustrates how to apply the item‐level ECA for attributes to estimate the correct classification rate of attributes patterns at the test level. Simulation results showed that two test‐level ECA indices, ECA_I_W (an index based on the independence assumption and the weighted sum of the item‐level ECAs) and ECA_C_M (an index based on Gaussian Copula function that incorporates the dependence structure of the events of attribute classification and the simple average of the item‐level ECAs), could make an accurate prediction for correct classification rates of attribute patterns.

show abstract

Attribute‐Level and Pattern‐Level Classification Consistency and Accuracy Indices for Cognitive Diagnostic Assessment

Abstract: Classification consistency and accuracy are viewed as important indicators for evaluating the reliability and validity of classification results in cognitive diagnostic assessment (CDA

Cited by 51 publications

References 35 publications

Diagnosing EFL learners’ writing ability: a diagnostic classification modeling analysis

Diagnosing EFL learners’ writing ability: a diagnostic classification modeling analysis

The Development of an Instrument for Longitudinal Learning Diagnosis of Rational Number Operations Based on Parallel Tests

An Item‐Level Expected Classification Accuracy and Its Applications in Cognitive Diagnostic Assessment

Contact Info

Product

Resources

About