Cognitive Diagnostic Models and Computerized Adaptive Testing: Two New Item-Selection Methods That Incorporate Response Times

Finkelman, Matthew; Kim, Wonsuk; Weissman, Alexander; Cook, Robert L.

doi:10.7333/1412-0204059

Cited by 12 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, MER performance might differ under practical constraints (e.g., item exposure control, test overlap control, or content balancing), so future studies can examine these phenomena. As information technology has enabled recording of response times, they can be incorporated into item selection methods (e.g., Finkelman et al, 2014), so future studies might incorporate response times into MER. As initial item selection can affect CAT efficiency (Xu et al, 2016), future studies can combine MER with optimal initial item selection.…”

Section: Discussionmentioning

confidence: 99%

Reducing the Misclassification Costs of Cognitive Diagnosis Computerized Adaptive Testing: Item Selection With Minimum Expected Risk

Hsu

Wang

2022

Applied Psychological Measurement

View full text Add to dashboard Cite

Cognitive diagnosis computerized adaptive testing (CD-CAT) aims to identify each examinee’s strengths and weaknesses on latent attributes for appropriate classification into an attribute profile. As the cost of a CD-CAT misclassification differs across user needs (e.g., remedial program vs. scholarship eligibilities), item selection can incorporate such costs to improve measurement efficiency. This study proposes such a method, minimum expected risk (MER), based on Bayesian decision theory. According to simulations, using MER to identify examinees with no mastery (MER-U0) or full mastery (MER-U1) showed greater classification accuracy and efficiency than other methods for these attribute profiles, especially for shorter tests or low quality item banks. For other attribute profiles, regardless of item quality or termination criterion, MER methods, modified posterior-weighted Kullback–Leibler information (MPWKL), posterior-weighted CDM discrimination index (PWCDI), and Shannon entropy (SHE) performed similarly and outperformed posterior-weighted attribute-level CDM discrimination index (PWACDI) in classification accuracy and test efficiency, especially on short tests. MER with a zero-one loss function, MER-U0, MER-U1, and PWACDI utilized item banks more effectively than the other methods. Overall, these results show the feasibility of using MER in CD-CAT to increase the accuracy for specific attribute profiles to address different user needs.

show abstract

Section: Discussionmentioning

confidence: 99%

Reducing the Misclassification Costs of Cognitive Diagnosis Computerized Adaptive Testing: Item Selection With Minimum Expected Risk

Hsu

Wang

2022

Applied Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…The Example of PhysTCriTS Item Appearance in CAT PhysTCriTS which implements CAT system is able to present the problems (questions) to the students in accordance with their ability as Finkelman et al, (2014) state that the next item will then be different, the difficulty index of the next items will change depending on the previous question answered. In addition, CAT is able to provide quick feedback for both student and teacher.…”

Section: Figurementioning

confidence: 99%

“…CAT is said to be the most important psychological assessment development. By using CAT, the difficulty index of the next items will change depending on the previous question answered (Finkelman et. al, 2014).…”

Section: Introductionmentioning

confidence: 99%

Developing IRT-Based Physics Critical Thinking Skill Test: A CAT to Answer 21st Century Challenge

Istiyono¹,

Dwandaru²,

Lede³

et al. 2019

INT J INSTRUCTION

View full text Add to dashboard Cite

The objective of this study was to develop Physics critical thinking skill test using computerized adaptive test (CAT) based on item response theory (IRT). This research was a development research using 4-D (define, design, develop, and disseminate). The content validity of the items was proven using Aiken's V. The test trial involved 252 students in Yogyakarta, Indonesia. The data were analysed according to partial credit model (PCM). The test reliability was estimated using PCM based on information function (IF) and standard error measurement (SEM), whereas the empirical validity was proven through INFIT MNSQ. The results showed that all items were valid with Aiken's V spread from 0.67 to 1.00 and INFIT MNSQ values from 0.86 to 1.20. The item difficulty index ranges from -0.75 to 1.30, which was a good item difficulty index. The IF and SEM showed that PhysTCriTS was suitable to measure students' critical thinking abilities from -1.80 to 1.50, with the reliability score reaches 0.75. The results of this study have an implication to reduce cheating in a test because each student gets a different item in accordance to the student's ability. The test using CAT may accurately and effectively measure physics critical thinking skill of students.

show abstract

“…In order to prevent differential speededness in high-stakes testing situations, this sensitivity of items to working speed is very useful in the test development process. Recently, Finkelman, Kim, Weissman, and Cook (2014) published a paper on item selection for cognitive diagnostic models and CAT in which RTs were taken into account.…”

Section: Introductionmentioning

confidence: 99%

Stochastic programming for individualized test assembly with mixture response time models

Veldkamp

Avetisyan

Weissman³

et al. 2017

Computers in Human Behavior

Self Cite

View full text Add to dashboard Cite

Cognitive Diagnostic Models and Computerized Adaptive Testing: Two New Item-Selection Methods That Incorporate Response Times

Cited by 12 publications

References 23 publications

Reducing the Misclassification Costs of Cognitive Diagnosis Computerized Adaptive Testing: Item Selection With Minimum Expected Risk

Reducing the Misclassification Costs of Cognitive Diagnosis Computerized Adaptive Testing: Item Selection With Minimum Expected Risk

Developing IRT-Based Physics Critical Thinking Skill Test: A CAT to Answer 21st Century Challenge

Stochastic programming for individualized test assembly with mixture response time models

Contact Info

Product

Resources

About