Hybrid Computerized Adaptive Testing: From Group Sequential Design to Fully Sequential Design

Abstract

Computerized adaptive testing (CAT) and multistage testing (MST) have become two of the most popular modes in large-scale computer-based sequential testing. Though most designs of CAT and MST exhibit strength and weakness in recent largescale implementations, there is no simple answer to the question of which design is better because different modes may fit different practical situations. This article proposes a hybrid adaptive framework to combine both CAT and MST, inspired by an analysis of the history of CAT and MST. The proposed procedure is a design which transitions from a group sequential design to a fully sequential design. This allows for the robustness of MST in early stages, but also shares the advantages of CAT in later stages with fine tuning of the ability estimator once its neighborhood has been identified. Simulation results showed that hybrid designs following our proposed principles provided comparable or even better estimation accuracy and efficiency than standard CAT and MST designs, especially for examinees at the two ends of the ability range.

Flexible Computerized Adaptive Tests to Detect Misconceptions and Estimate Ability Simultaneously

“…To address the aforementioned challenges, this study aims to propose a series of flexible CAT (FCAT) designs based on the SICM model. These FCAT designs are motivated by a hybrid adaptive framework proposed by S. Wang et al (2016).…”

A Review of Recent Advances in Adaptive Assessment

“…The item selection is performed automatically, but all stages of questions can be reviewed before administration (Chang, 2014). Wang et al (2016) suggest to ask a group of questions at the beginning of the test, when little information about learner ability is available, and progressively reduce the number of questions of each stage in order to increase opportunities to adapt. Also, asking questions in pools means that we can do content balancing at each stage, instead of jumping from one knowledge component to the other after every question.…”

On‐the‐Fly Constraint‐Controlled Assembly Methods for Multistage Adaptive Testing for Cognitive Diagnosis

“…Adaptive testing is a modern assessment method, which tailors the test based on each examinee's ability estimate and can make sure that each selected item provides as much information as possible for estimating the examinees’ ability (S. Wang, Lin, Chang, & Douglas, ). Compared with the traditional paper‐and‐pencil testing (P&P), adaptive testing is more efficient and the test length is much shorter without loss of measurement accuracy (Wainer & Eignor, ).…”

“…Zheng and Chang () called the MST that incorporates on‐the‐fly assembly strategies as the on‐the‐fly assembled multistage adaptive testing (OMST). Research (S. Wang et al., ; Zenisky, ; Zheng, Nozawa, Gao, & Chang, ) has indicated that MST has many advantages over CAT: first, MST adapts between stages but not within each stage, therefore examinees are allowed to navigate back and forth within current stage to review and change the answers which will make examinees feel less anxiety (Zenisky, ); second, MST allows test developers to adjust the number of stages and the module length assignment according to different situations; third, the nonstatistical constraints can be satisfied with ease in MST (Zheng et al, ). Due to these benefits, many high‐stakes tests have switched from the CAT mode to the MST mode (S. Wang et al, ).…”