Measurement Efficiency for Fixed-Precision Multidimensional Computerized Adaptive Tests: Comparing Health Measurement and Educational Testing Using Example Banks

Abstract

It is currently not entirely clear to what degree the research on multidimensional computerized adaptive testing (CAT) conducted in the field of educational testing can be generalized to fields such as health assessment, where CAT design factors differ considerably from those typically used in educational testing. In this study, the impact of a number of important design factors on CAT performance is systematically evaluated, using realistic example item banks for two main scenarios: health assessment (polytomous items, small to medium item bank sizes, high discrimination parameters) and educational testing (dichotomous items, large item banks, small- to medium-sized discrimination parameters). Measurement efficiency is evaluated for both between-item multidimensional CATs and separate unidimensional CATs for each latent dimension. In this study, we focus on fixed-precision (variable-length) CATs because it is both feasible and desirable in health settings, but so far most research regarding CAT has focused on fixed-length testing. This study shows that the benefits associated with fixed-precision multidimensional CAT hold under a wide variety of circumstances.

Practical Significance of Longitudinal Measurement Invariance Violations in the Dutch–Flemish PROMIS Item Banks for Depression and Anxiety: An Illustration With Ordered-Categorical Data

“…In addition to responsiveness, we suggest to study whether multidimensional computerized adaptive testing (MCAT; Paap, Born, & Braeken, 2019) with the Depression and Anxiety item banks can be more efficient and precise than CAT based on separate unidimensional item banks. In the current study, the item banks were treated as measurements of separate unidimensional constructs because PROMIS deliberately chose to develop their instruments in this way (Cella et al, 2007).…”

Making Fixed-Precision Between-Item Multidimensional Computerized Adaptive Tests Even Shorter by Reducing the Asymmetry Between Selection and Stopping Rules

“…This can be done in two ways, which we will label “hard restriction” and “soft restriction.” Hard restriction implies that, once the precision threshold δ has been met for dimension q , the remaining items loading on that dimension can no longer be selected throughout the CAT, which virtually results in a new item pool S k * (see, for example, Paap et al, 2018; Yao, 2013). Under soft restriction, only items can be selected that have a non-zero loading a id on at least one dimension that does not yet meet the precision threshold δ at the given iteration during the CAT administration (see, for example, Paap et al, 2019). Hence, the item pool is defined as:…”

“…Wang et al, 2019). By acknowledging the multidimensional dependence structure, MCAT typically results in more efficient tests as compared with using separate unidimensional CATs (UCATs) per dimension-a finding which has been shown to hold under a wide range of conditions (Paap et al, 2019).…”

Computerized adaptive testing in educational electronic environment of maritime higher education institutions

“…Analyzing recent research, we can see many scientists who have dealt with the problem of implementing adaptive tests in the educational process. Modern researchers related to the problem of adaptive testing include Albano et al [3], Austin et al [4], Cetin-Berber, Sari and Huggins-Manley [8], Collares and Cecilio-Fernandes [10], Istiyono et al [17], Kang, Zheng and Chang [18], Kozmina, Lukyantsev and Musorina [20], Lin and Chang [24], Paap, Born and Braeken [32], Samsudin, Somchut and Ismail [37], van der Linden and Choi [25], van der Linden and Ren [26], Wang et al [44], Yasuda et al [47].…”