Measuring Patient-Reported Outcomes Adaptively: Multidimensionality Matters!

Abstract

As there is currently a marked increase in the use of both unidimensional (UCAT) and multidimensional computerized adaptive testing (MCAT) in psychological and health measurement, the main aim of the present study is to assess the incremental value of using MCAT rather than separate UCATs for each dimension. Simulations are based on empirical data that could be considered typical for health measurement: a large number of dimensions (4), strong correlations among dimensions (.77-.87), and polytomously scored response data. Both variable- (SE < .316, SE < .387) and fixed-length conditions (total test length of 12, 20, or 32 items) are studied. The item parameters and variance–covariance matrix Φ are estimated with the multidimensional graded response model (GRM). Outcome variables include computerized adaptive test (CAT) length, root mean square error (RMSE), and bias. Both simulated and empirical latent trait distributions are used to sample vectors of true scores. MCATs were generally more efficient (in terms of test length) and more accurate (in terms of RMSE) than their UCAT counterparts. Absolute average bias was highest for variable-length UCATs with termination rule SE < .387. Test length of variable-length MCATs was on average 20% to 25% shorter than test length across separate UCATs. This study showed that there are clear advantages of using MCAT rather than UCAT in a setting typical for health measurement.

Making Fixed-Precision Between-Item Multidimensional Computerized Adaptive Tests Even Shorter by Reducing the Asymmetry Between Selection and Stopping Rules

“…The first approach we propose leaves the traditional item-selection criterion intact, but dynamically restricts the remaining item pool from which items can be selected. This can be done in two ways, which we will label “hard restriction” and “soft restriction.” Hard restriction implies that, once the precision threshold δ has been met for dimension q , the remaining items loading on that dimension can no longer be selected throughout the CAT, which virtually results in a new item pool S k * (see, for example, Paap et al, 2018; Yao, 2013). Under soft restriction, only items can be selected that have a non-zero loading a id on at least one dimension that does not yet meet the precision threshold δ at the given iteration during the CAT administration (see, for example, Paap et al, 2019).…”

“…Under hard restriction, the item pool restriction is permanent and not reversible, because it assumes an implied monotonic decrease of the marginal SE s during the CAT administration that is mathematically not strictly guaranteed. Note that both variants have been applied in specific studies as ad hoc solutions (see, for example, Paap et al, 2018; Yao, 2013), but their performance has not been formally evaluated or compared with other approaches.…”

“…An empirical multidimensional item bank of I = 194 items (4 or 5 ordinal response categories), designed to measure different aspects of quality of life (Paap et al, 2018), was used as a basis for the second study. Four dimensions were measured: fatigue (50 items), disease-specific complaints (46 items), physical function (63 items), and social roles and activities (35 items).…”

Construcción de un banco de ítems de facetas de neuroticismo para el desarrollo de un test adaptativo

“…La experiencia de algunos investigadores ha resultado prometedora, demostrando una ganancia incremental con respecto a los TAIs unidimensionales en la eficiencia de las mediciones (e.g. Makransky et al, 2013;Paap et al, 2017). No obstante, todavía se discuten aspectos vinculados a la determinación del algoritmo adaptativo como el método de selección de ítems (Smits, Paap, & Böhnke, 2018;Tu, Han, Cai & Gao, 2018) y criterios de interrupción (Wang, Chang, & Boughton, 2013;Yao, 2013).…”

Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes

“…These studies showed that multidimensional CATs were 25–33% shorter [ 41 – 43 ]. Two recent studies [ 44 , 45 ] in the context of health measurement showed that the efficiency gains reported for achievement measurement seem to generalize to health measurement: between-item multidimensional CATs were on average 20–38% shorter compared to using separate unidimensional CATs when between-dimension correlations were high ( r > .76). For weaker correlations ( r = .56), multidimensional CATs were on average 17% shorter than unidimensional CATs [ 45 ].…”