Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT

Chen, Ping; Xin, Tao; Wang, Chun; Chang, Hua Hua

doi:10.1007/s11336-012-9255-7

Cited by 46 publications

(85 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Because the number of responses to each new item will affect the calibration accuracy of the new item – and the new item with more responses tends to have higher calibration accuracy – to fairly calibrate all new items, the number of examinees answering each new item is controlled to be exactly equal to ( N × D )/ m (if ( N × D )/ m is an integer) or slightly more or less than ( N × D )/ m (if ( N × D )/ m is not an integer) as in the study of Chen et al . (). This can be accomplished by pre‐generating a random binary matrix T = ( T ij ) N × m with sums of each row all equal to D and sums of each column all equal to ( N × D )/ m .…”

Section: Simulation Designmentioning

confidence: 97%

Developing new online calibration methods for multidimensional computerized adaptive testing

Chen

Wang

Xin

et al. 2017

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

Multidimensional computerized adaptive testing (MCAT) has received increasing attention over the past few years in educational measurement. Like all other formats of CAT, item replenishment is an essential part of MCAT for its item bank maintenance and management, which governs retiring overexposed or obsolete items over time and replacing them with new ones. Moreover, calibration precision of the new items will directly affect the estimation accuracy of examinees' ability vectors. In unidimensional CAT (UCAT) and cognitive diagnostic CAT, online calibration techniques have been developed to effectively calibrate new items. However, there has been very little discussion of online calibration in MCAT in the literature. Thus, this paper proposes new online calibration methods for MCAT based upon some popular methods used in UCAT. Three representative methods, Method A, the 'one EM cycle' method and the 'multiple EM cycles' method, are generalized to MCAT. Three simulation studies were conducted to compare the three new methods by manipulating three factors (test length, item bank design, and level of correlation between coordinate dimensions). The results showed that all the new methods were able to recover the item parameters accurately, and the adaptive online calibration designs showed some improvements compared to the random design under most conditions.

show abstract

Section: Simulation Designmentioning

confidence: 97%

Developing new online calibration methods for multidimensional computerized adaptive testing

Chen

Wang

Xin

et al. 2017

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

show abstract

“…This index quantifies the estimation accuracy of the entire AMP (Chen, Xin, Wang, & Chang, ). Given simulated and estimated AMPs using MAP method, the PCCR is computed as

PCCR = \frac{1}{N} \sum_{i = 1}^{N} I ({\hat{α}}_{i} = α_{i}) .

…”

Section: Simulation Studymentioning

confidence: 99%

Attribute‐Level and Pattern‐Level Classification Consistency and Accuracy Indices for Cognitive Diagnostic Assessment

Wang

Song

Chen

et al. 2015

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

show abstract

“…The slip and guessing parameters were generated from a uniform distribution ranging between 0.05 and 0.25, and the same generated values were used for both the five-and seven-attribute conditions. These settings for the attribute-and item-parameter distributions were representative of those that are commonly observed in real data analyses, and were consistent with or similar to those used in previous studies (Chen, Xin, Wang, & Chang, 2012;de la Torre & Douglas, 2004;Hsu & Wang, 2015;Hsu et al, 2013;Huang & Wang, 2014;Kaplan et al, 2015;Mao & Xin, 2013;Wang, 2013). The Q-matrix was generated with reference to previous studies (Chen, Liu, & Ying, 2015;Chen et al, 2012), in which three basic matrices that specify all possible patterns for items measuring one, two and three attribute(s) were generated and then randomly reordered to constitute the full Q-matrix.…”

Section: Incorporating Rt Information Into Cd-catmentioning

confidence: 90%

Utilizing response times in cognitive diagnostic computerized adaptive testing under the higher‐order deterministic input, noisy ‘and’ gate model

Huang

2019

Brit J Math & Statis

View full text Add to dashboard Cite

Methods of cognitive diagnostic computerized adaptive testing (CD-CAT) under higherorder cognitive diagnosis models have been developed to simultaneously provide estimates of the attribute mastery statuses of examinees for formative assessment and estimates of a latent continuous trait for overall summative evaluation. In a typical CD-CAT environment, examinees are often subject to a time limit, and the examinees' response times (RTs) for specific test items can be routinely recorded by custom-made programs. Because examinees are individually administered tailored sets of test items from the item pool, they may experience different levels of speededness during testing and different levels of risk of running out of time. In this study, RTs were considered during the item-selection procedure to control the test speededness and the RTs were treated as useful information for improving latent trait estimation in CD-CAT under the higherorder deterministic input, noisy 'and' gate (DINA) model. A modified posterior-weighted Kullback-Leibler (PWKL) method that maximizes the item information per time unit and a shadow-test method that assembles a provisional test subject to a specified time constraint were developed. Two simulation studies were conducted to assess the effects of the proposed methods on the quality of CD-CAT for fixed-and variable-length exams. The results show that, compared with the traditional PWKL method, the proposed methods preserve a lower risk of running out of time while ensuring satisfactory attribute estimation and providing more accurate estimates of the latent trait and speed parameters. Finally, several suggestions for future research are proposed.

show abstract

Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT

Cited by 46 publications

References 19 publications

Developing new online calibration methods for multidimensional computerized adaptive testing

Developing new online calibration methods for multidimensional computerized adaptive testing

Attribute‐Level and Pattern‐Level Classification Consistency and Accuracy Indices for Cognitive Diagnostic Assessment

Utilizing response times in cognitive diagnostic computerized adaptive testing under the higher‐order deterministic input, noisy ‘and’ gate model

Contact Info

Product

Resources

About