A Comparison of Exposure Control Procedures in CAT Systems Based on Different Measurement Models for Testlets

Boyd, Aimee; Dodd, Barbara G.; Fitzpatrick, Steven J.

doi:10.1080/08957347.2013.765434

Cited by 13 publications

(13 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The randomesque procedure was used with both a group of three items (R-3) and a group of six items (R-6), which are typical group sizes investigated in the literature (Davis, 2004; Davis & Dodd, 2003; Davis & Dodd, 2008; Lee & Dodd, 2012; Leroux & Dodd, 2016). The PR-SE method had the maximum target exposure rate set to 0.30, which is a rate commonly used in previous studies especially with smaller polytomous item pools (Boyd et al, 2013; Davis et al, 2003; Lee & Dodd, 2012; Leroux & Dodd, 2016; Leroux et al, 2013; McClarty et al, 2006; Pastor et al, 2002; Revuelta & Ponsoda, 1998). As a baseline comparison, no exposure control (none) was investigated by selecting items solely based on maximum information.…”

Section: Methodsmentioning

confidence: 99%

“…All exposure control procedures in these four categories evaluated in CAT simulations have generally resulted in minor differences in measurement precision with polytomous items, but differed in the use of the items (Boyd, Dodd, & Fitzpatrick, 2013; Davis, 2004; Davis & Dodd, 2003, 2008; Davis, Pastor, Dodd, Chiang, & Fitzpatrick, 2003; Lee & Dodd, 2012; Leroux & Dodd, 2016; Pastor, Dodd, & Chang, 2002). The randomization and combination (specifically, progressive restricted) methods resulted in higher usage of the item bank than the other exposure control procedures.…”

Section: Exposure Control Techniques In Item Selectionmentioning

confidence: 99%

“…Within the randomization category, the randomesque and within 0.10 methods perform similarly with polytomous IRT models, but the randomesque method has been shown to slightly decrease the maximum exposure rate and amount of item overlap in comparison to the within 0.10 logits procedure (Davis, 2004; Davis & Dodd, 2008). The progressive restricted and PR-SE methods typically administer nearly the entire item pool, which is immensely more than the other procedures (Boyd et al, 2013; Lee & Dodd, 2012; Leroux & Dodd, 2016). The PR-SE procedure can be used with variable-length adaptive tests but the progressive restricted method is appropriate for fixed-length tests.…”

Section: Exposure Control Techniques In Item Selectionmentioning

confidence: 99%

See 2 more Smart Citations

An Investigation of Exposure Control Methods With Variable-Length CAT Using the Partial Credit Model

Leroux

Waid-Ebbs

Wen

et al. 2019

Applied Psychological Measurement

View full text Add to dashboard Cite

The purpose of this simulation study was to investigate the effect of several different item exposure control procedures in computerized adaptive testing (CAT) with variable-length stopping rules using the partial credit model. Previous simulation studies on CAT exposure control methods with polytomous items rarely considered variable-length tests. The four exposure control techniques examined were the randomesque with a group of three items, randomesque with a group of six items, progressive-restricted standard error (PR-SE), and no exposure control. The two variable-length stopping rules included were the SE and predicted standard error reduction (PSER), along with three item pools of varied sizes (43, 86, and 172 items). Descriptive statistics on number of nonconvergent cases, measurement precision, testing burden, item overlap, item exposure, and pool utilization were calculated. Results revealed that the PSER stopping rule administered fewer items on average while maintaining measurement precision similar to the SE stopping rule across the different item pool sizes and exposure controls. The PR-SE exposure control procedure surpassed the randomesque methods by further reducing test overlap, maintaining maximum exposure rates at the target rate or lower, and utilizing all items from the pool with a minimal increase in number of items administered and nonconvergent cases.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Exposure Control Techniques In Item Selectionmentioning

confidence: 99%

Section: Exposure Control Techniques In Item Selectionmentioning

confidence: 99%

See 1 more Smart Citation

An Investigation of Exposure Control Methods With Variable-Length CAT Using the Partial Credit Model

Leroux

Waid-Ebbs

Wen

et al. 2019

Applied Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…Despite the wide use of content balancing and exposure control methods in large-scale assessments, no prior research has been conducted that compares variable-length stopping rules under their implementation. CATs incorporating both exposure control and content balancing have been studied with fixed-length tests (Boyd, Dodd, & Fitzpatrick, 2013). There has also been research comparing exposure control methods using content balancing for dichotomous (i.e., Leroux et al, 2013) and polytomous (i.e., Davis, Pastor, Dodd, Chiang, & Fitzpatrick, 2003;Leroux & Dodd, 2016) IRT models that included separate FL and SEFL stopping rule conditions but did not compare variable-length stopping rules.…”

Section: Content Balancing and Exposure Controlmentioning

confidence: 99%

Comparing computer adaptive testing stopping rules under the generalized partial-credit model

et al. 2018

View full text Add to dashboard Cite

An important consideration of any computer adaptive testing (CAT) program is the criterion used for ending item administration-the stopping rule, which ensures that all examinees are assessed to the same standard. Although various stopping rules exist, none of them have been compared under the generalized partial-credit model (Muraki in Applied Psychological Measurement, 16, 159-176, 1992). In this simulation study we compared the performance of three variable-length stopping rules-standard error (SE), minimum information (MI), and change in theta (CT)-both in isolation and in combination with requirements of minimum and maximum numbers of items, as well as a fixed-length stopping rule. Each stopping rule was examined under two termination criteria-one a more lenient requirement (SE = 0.35, MI = 0.56, CT = 0.05), and one more stringent (SE = 0.30, MI = 0.42, CT = 0.02). The simulation design also included content-balancing and exposure controls, aspects of CAT that have been excluded in previous research comparing variable-length stopping rules. The minimum-information stopping rule produced biased theta estimates and varied greatly in measurement quality across the theta distribution. The absolute-change-in-theta stopping rule had strong performance when paired with a lower criterion and a minimum test length. The standard error stopping rule consistently provided the best balance of measurement precision and operational efficiency and was based on the fewest number of administered items necessary to obtain accurate and precise theta estimates, particularly when it was paired with a maximum-number-of-items stopping rule.

show abstract

“…In brief, when a method for controlling item exposure is used in combination with different item selection methods, item pools will yield different results based on how they are used depending on the algorithm used in that particular item selection method (Han, 2012 While focusing on controlling item exposure, the measurement precision should not be ignored. Item exposure control methods must balance the test security with the measurement precision (Boyd, 2003;Boyd, Dodd, & Fitzpatrick, 2013). For these reasons, the precision of ability level estimations and the utilization of item pool should be considered together when comparing the methods for controlling item exposure.…”

mentioning

confidence: 99%

Investigating Item Exposure Control Methods in Computerized Adaptive Testing

Öztürk¹,

Doğan²

2015

EDUC SCI-THEOR PRACT

View full text Add to dashboard Cite

This study aims to investigate the effects of item exposure control methods on measurement precision and on test security under various item selection methods and item pool characteristics. In this study, the Randomesque (with item group sizes of 5 and 10), Sympson-Hetter, and Fade-Away methods were used as item exposure control methods. Moreover, in order to establish a comparison baseline for other methods, the noexposure control condition was also included in the research. As item selection methods, Maximum Fisher Information, a-Stratification, and Gradual Maximum Information Ratio were employed. While a parameters were generated from a uniform distribution ranging between 0.50 and 2.00 and c parameters were generated from a uniform distribution ranging between 0.05 and 0.20 for both item pools, b parameters were generated from a uniform distribution ranging between -3.00 and +3.00 for medium difficulty item pool, and from a standard normal N(2, 1.5) distribution for high difficulty item pool. Based on the research findings, there were no great differences between the values of indicators used in determining measurement precision when the item exposure control methods were used. On the other hand, in terms of test security, it was found that in general, the Fade-Away Method for controlling item exposure yielded better results than did the other methods in reducing the skewness of item pool utilization and the test overlap.

show abstract

A Comparison of Exposure Control Procedures in CAT Systems Based on Different Measurement Models for Testlets

Cited by 13 publications

References 19 publications

An Investigation of Exposure Control Methods With Variable-Length CAT Using the Partial Credit Model

An Investigation of Exposure Control Methods With Variable-Length CAT Using the Partial Credit Model

Comparing computer adaptive testing stopping rules under the generalized partial-credit model

Investigating Item Exposure Control Methods in Computerized Adaptive Testing

Contact Info

Product

Resources

About