A New Online Calibration Method for Multidimensional Computerized Adaptive Testing

Chen, Ping; Wang, Chun

doi:10.1007/s11336-015-9482-9

Cited by 28 publications

(53 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We found that MM‐MEM‐AMC significantly improved the calibration accuracy and efficiency compared to M‐MEM‐AMC. A possible reason is that the K nodes drawn from M‐OEM‐AMC are already accurate enough, and an inaccurate provisional estimate of

Δ_{j}^{false(t false)}

brings more noise when updating

N ({boldθ}_{i}^{j} | {\overset{θ}{true}}_{i}^{j^{(t + 1)}}, {\overset{Ω}{true}}_{i}^{j^{(t + 1)}})

because M‐MEM often performs worse than M‐OEM especially for medium and large‐sized correlations between coordinate dimensions (Chen & Wang, ). Hence it is not worthwhile to keep drawing new sets of random nodes adaptively from the updated distribution

N ({boldθ}_{i}^{j} | {\overset{θ}{true}}_{i}^{j^{(t + 1)}}, {\overset{Ω}{true}}_{i}^{j^{(t + 1)}})

, which obviously can be contaminated by the errors in

Δ_{j}^{false(t false)}

.…”

Section: Online Calibration Methods In Mcatmentioning

confidence: 99%

Developing new online calibration methods for multidimensional computerized adaptive testing

Chen

Wang

Xin

et al. 2017

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

Multidimensional computerized adaptive testing (MCAT) has received increasing attention over the past few years in educational measurement. Like all other formats of CAT, item replenishment is an essential part of MCAT for its item bank maintenance and management, which governs retiring overexposed or obsolete items over time and replacing them with new ones. Moreover, calibration precision of the new items will directly affect the estimation accuracy of examinees' ability vectors. In unidimensional CAT (UCAT) and cognitive diagnostic CAT, online calibration techniques have been developed to effectively calibrate new items. However, there has been very little discussion of online calibration in MCAT in the literature. Thus, this paper proposes new online calibration methods for MCAT based upon some popular methods used in UCAT. Three representative methods, Method A, the 'one EM cycle' method and the 'multiple EM cycles' method, are generalized to MCAT. Three simulation studies were conducted to compare the three new methods by manipulating three factors (test length, item bank design, and level of correlation between coordinate dimensions). The results showed that all the new methods were able to recover the item parameters accurately, and the adaptive online calibration designs showed some improvements compared to the random design under most conditions.

show abstract

Δ_{j}^{false(t false)}

brings more noise when updating

N ({boldθ}_{i}^{j} | {\overset{θ}{true}}_{i}^{j^{(t + 1)}}, {\overset{Ω}{true}}_{i}^{j^{(t + 1)}})

N ({boldθ}_{i}^{j} | {\overset{θ}{true}}_{i}^{j^{(t + 1)}}, {\overset{Ω}{true}}_{i}^{j^{(t + 1)}})

, which obviously can be contaminated by the errors in

Δ_{j}^{false(t false)}

.…”

Section: Online Calibration Methods In Mcatmentioning

confidence: 99%

Developing new online calibration methods for multidimensional computerized adaptive testing

Chen

Wang

Xin

et al. 2017

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

show abstract

“…The number of examinees who answer each new item must be sufficiently large to provide accurate item parameter estimates without placing an undue burden on examinees (Wainer and Mislevy, 1990). This paper investigates one sample size (3,000) and assumes that each examinee answers 5 new items, thus the number of examinees who answer each new item is approximately 750 [(3,000×5)/20] on average as in previous studies (e.g., Chen et al, 2012;Chen and Wang, 2016;He et al, 2017). In Study 1, the number of examinees to each new item is set 700.…”

Section: Simulation Detailsmentioning

confidence: 99%

Online Calibration of Polytomous Items Under the Graded Response Model

et al. 2020

View full text Add to dashboard Cite

Computerized adaptive testing (CAT) is an efficient testing mode, which allows each examinee to answer appropriate items according his or her latent trait level. The implementation of CAT requires a large-scale item pool, and item pool needs to be frequently replenished with new items to ensure test validity and security. Online calibration is a technique to calibrate the parameters of new items in CAT, which seeds new items in the process of answering operational items, and estimates the parameters of new items through the response data of examinees on new items. The most popular estimation methods include one EM cycle method (OEM) and multiple EM cycle method (MEM) under dichotomous item response theory models. This paper extends OEM and MEM to the graded response model (GRM), a popular model for polytomous data with ordered categories. Two simulation studies were carried out to explore online calibration under a variety of conditions, including calibration design, initial item parameter calculation methods, calibration methods, calibration sample size and the number of categories. Results show that the calibration accuracy of new items were acceptable, and which were affected by the interaction of some factors, therefore some conclusions were given.

show abstract

“…where E(a r ) = exp(rm + (r 2 s 2 =2)) and F(Á) is the cumulative distribution function of standard normal distribution (P. Chen & Wang 2016;Lien, 1985). Thus, m a ¼ D E(aj0:2\a\2) = 0:8804 and var(a) ¼ D var(aj0:2\a\2) = 0:2296 can be obtained by setting L a = 0:2, U a = 2, m = 0, and s = 1.…”

Section: Generation Of Items and Examineesmentioning

confidence: 99%

A New Online Calibration Method Based on Lord’s Bias-Correction

Chen

et al. 2017

Applied Psychological Measurement

Self Cite

View full text Add to dashboard Cite

Online calibration technique has been widely employed to calibrate new items due to its advantages. Method A is the simplest online calibration method and has attracted many attentions from researchers recently. However, a key assumption of Method A is that it treats personparameter estimatesûs (obtained by maximum likelihood estimation [MLE]) as their true values us, thus the deviation of the estimatedûs from their true values might yield inaccurate item calibration when the deviation is nonignorable. To improve the performance of Method A, a new method, MLE-LBCI-Method A, is proposed. This new method combines a modified Lord's biascorrection method (named as maximum likelihood estimation-Lord's bias-correction with iteration [MLE-LBCI]) with the original Method A in an effort to correct the deviation ofûs which may adversely affect the item calibration precision. Two simulation studies were carried out to explore the performance of both MLE-LBCI and MLE-LBCI-Method A under several scenarios. Simulation results showed that MLE-LBCI could make a significant improvement over the ML ability estimates, and MLE-LBCI-Method A did outperform Method A in almost all experimental conditions.

show abstract

A New Online Calibration Method for Multidimensional Computerized Adaptive Testing

Cited by 28 publications

References 30 publications

Developing new online calibration methods for multidimensional computerized adaptive testing

Developing new online calibration methods for multidimensional computerized adaptive testing

Online Calibration of Polytomous Items Under the Graded Response Model

A New Online Calibration Method Based on Lord’s Bias-Correction

Contact Info

Product

Resources

About