Statistical Applications in Educational Measurement

Chang, Hua Hua; Chun, Wang; Zhang, Susu

doi:10.1146/annurev-statistics-042720-104044

Cited by 10 publications

(9 citation statements)

References 103 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A non‐sequential implementation can be found in Chalmers and Ng ( 2017 ). Finally, while the proposed method is aimed at detecting item compromise occurring in large‐scale continuous tests, as remote learning and testing become increasingly relevant, future studies can also explore the adaptation of the methods here to address the item compromise issues in formative classroom assessments (Chang et al, 2021 ).…”

Section: Discussionmentioning

confidence: 99%

Compromised item detection: A Bayesian change‐point perspective

Zhang

Chang

2022

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

Psychometric methods for accurate and timely detection of item compromise have been a long-standing topic. While Bayesian methods can incorporate prior knowledge or expert inputs as additional information for item compromise detection, they have not been employed in item compromise detection itself. The current study proposes a two-phase Bayesian change-point framework for both stationary and real-time detection of changes in each item's compromise status. In Phase I, a stationary Bayesian change-point model for compromise detection is fitted to the observed responses over a specified time-frame. The model produces parameter estimates for the change-point of each item from uncompromised to compromised, as well as structural parameters accounting for the post-change response distribution. Using the post-change model identified in Phase I, the Shiryaev procedure for sequential testing is employed in Phase II for real-time monitoring of item compromise. The proposed methods are evaluated in terms of parameter recovery, detection accuracy, and detection efficiency under various simulation conditions and in a real data example. The proposed method also showed superior detection accuracy and efficiency compared to the cumulative sum procedure.

show abstract

Section: Discussionmentioning

confidence: 99%

Compromised item detection: A Bayesian change‐point perspective

Zhang

Chang

2022

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

show abstract

“…CDMs are a class of psychometric models that combine modern statistical methods with cognitive theories and therefore produce feedbacks that reflect the cognitive and psychological characteristics of the subjects (Templin and Henson, 2010;Wu et al, 2020Wu et al, , 2021c. It holds great promise in providing fine-grained feedback (Leighton and Gierl, 2007;Templin and Bradshaw, 2014;Chang et al, 2021). For diagnostic purposes, CDMs could identify multiple criterion-referenced interpretations for numerous attributes in solving the test items.…”

Section: Assessments Of Cognitive Diagnostic Models Via Large-scale Datasetsmentioning

confidence: 99%

An International Comparative Study of Students' Scientific Explanation Based on Cognitive Diagnostic Assessment

Yang

et al. 2021

Front. Psychol.

View full text Add to dashboard Cite

Scientific explanation is one of the most core concepts in science education, and its mastery level is crucial for a deep understanding of the nature of science. As a new generation of assessment theory, cognitive diagnostic assessment (CDA) can get the knowledge of students' mastery of fine-grained knowledge. Based on the extant research, this research has formed eight attributes of scientific explanation concepts. By coding the Trends in International Mathematics and Science Study (TIMSS) test items, a CAD tool was formed. Data collected from 574 Grade 4 students in Hangzhou, China, combined with the data of the United States, Singapore, Australia, the United Kingdom, and Russia, were used in our study. The Deterministic Inputs, Noisy “And” gate (DINA) model was used to analyze the results from three aspects: the probability of mastery of attributes, the international comparison of knowledge states, and the analysis of learning paths. This study provided a new perspective from a CDA approach on the assessment of scientific explanation.

show abstract

“…This research identified several common reading attributes crucial for successful reading and supported the use of CDMs as a promising approach that pinpoints students’ weaknesses and strengths in mastering these reading attributes (Li et al, 2021). However, the application of such fine-grained feedback is scarce as it requires students to answer a static, time-consuming, or less accurate test paper (Chang et al, 2021). Formative assessment of children’s reading ability and subskills is essential and ongoing throughout their primary school years (Carlson et al, 2014), and therefore, it is worthy of exploring a more accurate and efficient way.…”

Section: Introductionmentioning

confidence: 99%

Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?

Huang²,

Liu

2023

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

Cognitive diagnostic computerized adaptive testing (CD-CAT) is a cutting-edge technology in educational measurement that targets at providing feedback on examinees’ strengths and weaknesses while increasing test accuracy and efficiency. To date, most CD-CAT studies have made methodological progress under simulated conditions, but little has applied CD-CAT to real educational assessment. The present study developed a Chinese reading comprehension item bank tapping into six validated reading attributes, with 195 items calibrated using data of 28,485 second to sixth graders and the item-level cognitive diagnostic models (CDMs). The measurement precision and efficiency of the reading CD-CAT system were compared and optimized in terms of crucial CD-CAT settings, including the CDMs for calibration, item selection methods, and termination rules. The study identified seven dominant reading attribute mastery profiles that stably exist across grades. These major clusters of readers and their variety with grade indicated some sort of reading developmental mechanisms that advance and deepen step by step at the primary school level. Results also suggested that compared to traditional linear tests, CD-CAT significantly improved the classification accuracy without imposing much testing burden. These findings may elucidate the multifaceted nature and possible learning paths of reading and raise the question of whether CD-CAT is applicable to other educational domains where there is a need to provide formative and fine-grained feedback but where there is a limited amount of test time.

show abstract

Statistical Applications in Educational Measurement

Cited by 10 publications

References 103 publications

Compromised item detection: A Bayesian change‐point perspective

Compromised item detection: A Bayesian change‐point perspective

An International Comparative Study of Students' Scientific Explanation Based on Cognitive Diagnostic Assessment

Diagnosing Primary Students’ Reading Progression: Is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?

Contact Info

Product

Resources

About