Item usage in a multidimensional computerized adaptive test (MCAT) measuring health-related quality of life

Abstract

PurposeExamining item usage is an important step in evaluating the performance of a computerized adaptive test (CAT). We study item usage for a newly developed multidimensional CAT which draws items from three PROMIS domains, as well as a disease-specific one.MethodsThe multidimensional item bank used in the current study contained 194 items from four domains: the PROMIS domains fatigue, physical function, and ability to participate in social roles and activities, and a disease-specific domain (the COPD-SIB). The item bank was calibrated using the multidimensional graded response model and data of 795 patients with chronic obstructive pulmonary disease. To evaluate the item usage rates of all individual items in our item bank, CAT simulations were performed on responses generated based on a multivariate uniform distribution. The outcome variables included active bank size and item overuse (usage rate larger than the expected item usage rate).ResultsFor average θ-values, the overall active bank size was 9–10%; this number quickly increased as θ-values became more extreme. For values of −2 and +2, the overall active bank size equaled 39–40%. There was 78% overlap between overused items and active bank size for average θ-values. For more extreme θ-values, the overused items made up a much smaller part of the active bank size: here the overlap was only 35%.ConclusionsOur results strengthen the claim that relatively short item banks may suffice when using polytomous items (and no content constraints/exposure control mechanisms), especially when using MCAT.Electronic supplementary materialThe online version of this article (doi:10.1007/s11136-017-1624-3) contains supplementary material, which is available to authorized users.

Measurement Efficiency for Fixed-Precision Multidimensional Computerized Adaptive Tests: Comparing Health Measurement and Educational Testing Using Example Banks

“…The first studies comparing MCAT with UCAT performance focused on measuring ability; these studies showed that fixed-length MCATs were 25% to 33% shorter and resulted in more accurate ability estimates (Li & Schafer, 2005; Luecht, 1996; Segall, 1996). A recent study comparing MCAT with UCAT in the context of health measurement showed that MCATs were, on average, 20% to 25% shorter compared with using separate UCATs (Paap, Kroeze, Glas, et al, 2017). Although it may be tempting, it is difficult to compare these figures across studies directly.…”

“…This was not entirely surprising, given the dichotomous nature of the items. When it comes to polytomous items, previous studies have shown that item banks as small as 20 to 30 items per dimension may be adequate in supporting CAT when exposure control is not an issue (Boyd, Dodd, & Choi, 2010; Dodd, De Ayala, & Koch, 1995; Paap, Kroeze, Terwee, van der Palen, & Veldkamp, 2017). Overall, this study showed support for these findings, but it also allowed for refinement: Feasibility in the HEPO scenario was dependent on the observed θ-score pattern (for banks with 30 items or fewer per dimension).…”

“…Three of these cells concerned the health (HEDI or HEPO)/high correlation/discordant θ-score group combination. Some authors have speculated that MCAT may have little to add if polytomous items are used, because polytomous items are generally considerably richer in information than dichotomous items (e.g., Paap, Kroeze, Glas, et al, 2017). Our results did not support this conjecture: Although UCATs based on polytomous items were already very short, MCATs were shorter still.…”

A new approach to assessing shyness of college students using computerized adaptive testing: CAT-Shyness

“…After investigating a large number of studies, the above two models were not only commonly used polytomously scoring models in IRT, but also commonly used in CAT (e.g. Paap, Kroeze, Terwee, Palen, & Veldkamp, 2017). Therefore, the model with the smaller test-fit indices between the GRM and the GPCM was selected for further analysis.…”

Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes

“…Therefore the presented item bank should be large enough to support adequate measurement precision for all relevant levels of the latent construct one wishes to assess in the given application [ 48 , 49 ]. Past studies suggest that item banks as small as 20–30 polytomous items per dimension may be adequate when exposure control is not an issue [ 50 , 51 ]. A recent study [ 45 ] found that the required item bank size depended on adequate targeting (the match between the item bank and the latent trait distribution).…”