Some recommendations for developing multidimensional computerized adaptive tests for patient-reported outcomes

Abstract

PurposeMultidimensional item response theory and computerized adaptive testing (CAT) are increasingly used in mental health, quality of life (QoL), and patient-reported outcome measurement. Although multidimensional assessment techniques hold promises, they are more challenging in their application than unidimensional ones. The authors comment on minimal standards when developing multidimensional CATs.MethodsPrompted by pioneering papers published in QLR, the authors reflect on existing guidance and discussions from different psychometric communities, including guidelines developed for unidimensional CATs in the PROMIS project.ResultsThe commentary focuses on two key topics: (1) the design, evaluation, and calibration of multidimensional item banks and (2) how to study the efficiency and precision of a multidimensional item bank. The authors suggest that the development of a carefully designed and calibrated item bank encompasses a construction phase and a psychometric phase. With respect to efficiency and precision, item banks should be large enough to provide adequate precision over the full range of the latent constructs. Therefore CAT performance should be studied as a function of the latent constructs and with reference to relevant benchmarks. Solutions are also suggested for simulation studies using real data, which often result in too optimistic evaluations of an item bank’s efficiency and precision.DiscussionMultidimensional CAT applications are promising but complex statistical assessment tools which necessitate detailed theoretical frameworks and methodological scrutiny when testing their appropriateness for practical applications. The authors advise researchers to evaluate item banks with a broad set of methods, describe their choices in detail, and substantiate their approach for validation.

Measurement Efficiency for Fixed-Precision Multidimensional Computerized Adaptive Tests: Comparing Health Measurement and Educational Testing Using Example Banks

“…Currently, most CATs used for health measurement are based on unidimensional IRT models. However, multidimensional IRT (MIRT) models (Reckase, 2009) and multidimensional CAT (MCAT; Luecht, 1996;Segall, 1996Segall, , 2010 are becoming increasingly popular in health measurement; this is especially true for between-item multidimensional models (Smits, Paap, & Boehnke, 2018). In recent years, several authors have shown that taking into account the correlation among health-related dimensions, when estimating patient scores in CATs, offers potential benefits (e.g., Nikolaus et al, 2015;Paap, Kroeze, Glas, et al, 2018).…”

Symptom burden and health-related quality of life in chronic kidney disease: A global systematic review and meta-analysis

“…Such PROM ‘item burden’ has been widely recognised as an important threat to adherence [ 23 ]. Given the high number of symptoms experienced by patients with CKD, the use of contemporary psychometrics, encompassing item response theory (IRT) and computerised adaptive testing (CAT), may be warranted in order to develop new measures that capture sufficient information regarding all patient-important symptoms, whilst minimising questionnaire burden [ 24 ]. CATs efficiently select questions from an IRT-calibrated item bank that are targeted to an individual’s ability/trait level using an adaptive algorithm, minimising the number of items administered, for example, the Patient-Reported Outcomes Measurement Information System (PROMIS) physical function CAT [ 25 – 27 ].…”

The Use of Patient-Reported Outcomes in Patients Treated With Maintenance Hemodialysis: A Perspective

“…The National Institutes of Health PROMIS project uses item response theory (IRT) and computer adaptive technology (CAT) to capture global aspects of HRQoL and has been shown to be a potentially valuable tool to assess impact of chronic kidney disease in paediatric populations (62). CAT simplifies PRO completion by selecting questions with patient-specific relevance This process has a high level of reliability, and low burden, as patient responses are used to guide subsequent questions (63) Further research is required to identify/develop optimum PROs for routine use in HD settings. Cultural appropriateness and cross cultural reliability of such PROs should be assessed, with issues such as spirituality, which have different resonance for different ethnic groups (64) being considered.…”

“…Platforms which can collect, collate and feedback PRO data, via the electronic health record /national renal registries, need further investigation and investment. Development of digital systems using item banks and CAT algorithms could be one way of lessening the burden and allow PRO data collected at specific time points to be used for multiple purposes (63).…”