Statistical Foundations for Computerized Adaptive Testing with Response Revision

Wang, Shiyu; Fellouris, Georgios; Chang, Hua‐Hua

doi:10.1007/s11336-019-09662-9

Cited by 5 publications

(16 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address the above concerns, researchers have proposed several methods to allow item review in CAT from different perspectives (Bowles & Pommerich, 2001; Han, 2013; Papanastasiou & Reckase, 2007; Stocking, 1997; S. Wang et al, 2019; Yen et al, 2012).…”

Section: Representative Methods For Reviewable Catmentioning

confidence: 99%

The Block Item Pocket Method for Reviewable Multidimensional Computerized Adaptive Testing

Zhe

Chen

Xin

2020

Applied Psychological Measurement

View full text Add to dashboard Cite

Most computerized adaptive testing (CAT) programs do not allow item review due to a decrease in estimation precision and aberrant manipulation strategies. In this article, a block item pocket (BIP) method that combines the item pocket method with the successive block method to realize reviewable CAT was proposed. A worst-case but still reasonable answering strategy and the Wainer-like manipulation strategy were simulated to evaluate the estimation precision of reviewable unidimensional computerized adaptive testing (UCAT) and multidimensional computerized adaptive testing (MCAT) under a series of BIP settings. For both UCAT and MCAT, it was found that the estimation precision of the BIP method improved as the number of blocks increased or the item pocket size decreased under the reasonable strategy. The BIP method was more effective in handling the Wainer-like strategy. With the help of block design, the BIP method can still maintain acceptable estimation precision under slightly large total IP size conditions. These results suggested that the BIP method was a reliable solution for both reviewable UCAT and MCAT.

show abstract

Section: Representative Methods For Reviewable Catmentioning

confidence: 99%

The Block Item Pocket Method for Reviewable Multidimensional Computerized Adaptive Testing

Zhe

Chen

Xin

2020

Applied Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…Hence, changing responses retrospectively may impact the measurement precision, which results in larger standard errors [69,[73][74][75][76][77][78]. Therefore, allowing item revision within CATs has been controversially discussed in the literature, even if some contributions encountered this measurement problem (see, e.g., [66,77,[79][80][81][82]). While it can be argued that only a few persons might change their responses [83], a lack of this ability appears to contribute to increased test anxiety.…”

Section: Test Anxietymentioning

confidence: 99%

Item Parameter Estimation in Multistage Designs: A Comparison of Different Estimation Approaches for the Rasch Model

Steinfeld

Robitzsch

2021

Psych

View full text Add to dashboard Cite

There is some debate in the psychometric literature about item parameter estimation in multistage designs. It is occasionally argued that the conditional maximum likelihood (CML) method is superior to the marginal maximum likelihood method (MML) because no assumptions have to be made about the trait distribution. However, CML estimation in its original formulation leads to biased item parameter estimates. Zwitser and Maris (2015, Psychometrika) proposed a modified conditional maximum likelihood estimation method for multistage designs that provides practically unbiased item parameter estimates. In this article, the differences between different estimation approaches for multistage designs were investigated in a simulation study. Four different estimation conditions (CML, CML estimation with the consideration of the respective MST design, MML with the assumption of a normal distribution, and MML with log-linear smoothing) were examined using a simulation study, considering different multistage designs, number of items, sample size, and trait distributions. The results showed that in the case of the substantial violation of the normal distribution, the CML method seemed to be preferable to MML estimation employing a misspecified normal trait distribution, especially if the number of items and sample size increased. However, MML estimation using log-linear smoothing lea to results that were very similar to the CML method with the consideration of the respective MST design.

show abstract

“…In this way, Wang et al (2017) showed that the resulting ability estimator preserves the asymptotic efficiency of the conventional CAT when the number of revisions is small relative to the number of items. Wang et al (2019) further reformulated this conditional estimation approach and extended the theoretical results to a larger class of IRT models. In addition, an equal-weight estimation approach, which is an ad hoc modification of the original Wang et al (2017) conditional estimation method, was proposed to deal with response revision with dichotomous IRT models.…”

Section: Introductionmentioning

confidence: 99%

“…These two ability estimation methods in Wang et al (2017, 2019) essentially assign a weight to each distinct answer of the same item in a deterministic way. For the conditional estimation method, the weight for the initial weight and any subsequent revisions is determined automatically through the score function, and the initial answer always carries a larger weight compared with the revisions.…”

Section: Introductionmentioning

confidence: 99%

“…Our proposed method determines the “weight” for each distinct response adaptively based on the compatibility of each observation with the assumed statistical model in relation to the remaining observations. This proposed estimation scheme can be viewed as a generalization of the ability estimation methods in Wang et al (2017, 2019). Two types of weight estimation approaches are proposed in Section 3.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Adaptive Weight Estimation of Latent Ability: Application to Computerized Adaptive Testing With Response Revision

Wang¹,

Xiao

Cohen³

2020

Journal of Educational and Behavioral Statistics

Self Cite

View full text Add to dashboard Cite

An adaptive weight estimation approach is proposed to provide robust latent ability estimation in computerized adaptive testing (CAT) with response revision. This approach assigns different weights to each distinct response to the same item when response revision is allowed in CAT. Two types of weight estimation procedures, nonfunctional and functional weight, are proposed to determine the weight adaptively based on the compatibility of each revised response with the assumed statistical model in relation to remaining observations. The application of this estimation approach to a data set collected from a large-scale multistage adaptive testing demonstrates the capability of this method to reveal more information regarding the test taker’s latent ability by using the valid response path compared with only using the very last response. Limited simulation studies were concluded to evaluate the proposed ability estimation method and to compare it with several other estimation procedures in literature. Results indicate that the proposed ability estimation approach is able to provide robust estimation results in two test-taking scenarios.

show abstract

Statistical Foundations for Computerized Adaptive Testing with Response Revision

Cited by 5 publications

References 28 publications

The Block Item Pocket Method for Reviewable Multidimensional Computerized Adaptive Testing

The Block Item Pocket Method for Reviewable Multidimensional Computerized Adaptive Testing

Item Parameter Estimation in Multistage Designs: A Comparison of Different Estimation Approaches for the Rasch Model

Adaptive Weight Estimation of Latent Ability: Application to Computerized Adaptive Testing With Response Revision

Contact Info

Product

Resources

About