A New Stopping Rule for Computerized Adaptive Testing

Abstract

The goal of the current study was to introduce a new stopping rule for computerized adaptive testing. The predicted standard error reduction stopping rule (PSER) uses the predictive posterior variance to determine the reduction in standard error that would result from the administration of additional items. The performance of the PSER was compared to that of the minimum standard error stopping rule and a modified version of the minimum information stopping rule in a series of simulated adaptive tests, drawn from a number of item pools. Results indicate that the PSER makes efficient use of CAT item pools, administering fewer items when predictive gains in information are small and increasing measurement precision when information is abundant. KeywordsComputerized Adaptive Testing (CAT); Stopping Rule; Testing Burden According to Wainer (2000) an adaptive test can be considered complete after a predetermined number of items have been administered, when a predetermined level of measurement precision has been reached, or when a predetermined length of time has elapsed. The two most commonly used methods for determining when a computerized adaptive test is complete are the fixed length and variable length stopping rules.Under a fixed length stopping rule, an adaptive test is terminated when a predetermined number of items have been administered. Accordingly, all examinees are administered the same number of items, regardless of the degree of measurement precision achieved upon termination of the test. The primary advantage of the fixed length stopping rule is its simplicity. However, one consequence of the implementation of the fixed length stopping rule is that examinees will be measured with different degrees of precision, with larger measurement error typically occurring at extreme trait levels. Additionally, a fixed length stopping rule may limit the efficiency of an adaptive test through the unnecessary administration of items that contribute little information about examinee trait level.In contrast, variable length stopping rules typically seek to achieve a certain degree of measurement precision for all examinees, even when doing so means that some examinees are given more items than others. Two types of variable length stopping rules have been Send correspondence to: Seung W. Choi, s-choi@northwestern.edu, Northwestern University Feinberg School of Medicine, Department of Medical Social Sciences, 710 N. Lake Shore Dr., Chicago, IL 60611, USA, Phone: (312) Fax: (312) 503-9800. NIH Public Access Author ManuscriptEduc Psychol Meas. Author manuscript; available in PMC 2011 January 27. NIH-PA Author ManuscriptNIH-PA Author Manuscript NIH-PA Author Manuscript used (Dodd, Koch, & De Ayala, 1993). These are the standard error (SE) stopping rule and the minimum information stopping rule. Of these, the most commonly used has been the SE stopping rule, which terminates an adaptive test when a predetermined standard error has been reached for the most recent examinee trait estimate (Boyd, Dodd, & Choi, 2010)....

Quantitative Psychology Research

“…The administered tests for certain examinees may be undesirably lengthy or short because the required precision cannot be met or few items have improved the precision significantly. Under the unidimensional framework, some studies have been done on using different stopping rules in CAT (Dodd et al 1993), such as the minimum standard error (SE) stopping rule, the minimum information stopping rule (Dodd et al 1989), and the predicted standard error reduction (PSER) stopping rule (Choi et al 2011). Under the multidimensional framework, many previous studies on item selection methods were conducted in fixed-length MCATs (Mulder and van der Linden 2009;Su and Huang 2014;Wang et al 2011a, b;Yao 2011Yao , 2012.…”

Enhancing a Short Measure of Big Five Personality Traits With Bayesian Scaling

“…When adaptive testing is based on an IRT model, variable length stop rules are typically based on either standard error or minimum information approaches (Choi, Grady, & Dodd, 2011). In this Bayesian application, however, the premise is that the stop rule will vary by the intent of the assessment and preferences of the clinician or researcher.…”

Computerized Adaptive Test (CAT) Applications and Item Response Theory Models for Polytomous Items

“…For this reason, Choi, Grady and Dodd (2011) proposed an alternative to the standard error and information function based methods, Predicted Standard Error Reduction (PSER). In this method, if a new item is applied, it is estimated how much reduction will occur in the standard error, and if the decrease amount is below a predetermined value, the test is stopped.…”