Development and Validation of a Computerized Adaptive Assessment Tool for Discrimination and Measurement of Psychotic Symptoms

Abstract

Objective Time constraints limit the use of measurement-based approaches in research and routine clinical management of psychosis. Computerized adaptive testing (CAT) can reduce administration time, thus increasing measurement efficiency. This study aimed to develop and test the capacity of the CAT-Psychosis battery, both self-administered and rater-administered, to measure the severity of psychotic symptoms and discriminate psychosis from healthy controls. Methods An item bank was developed and calibrated. Two raters administered CAT-Psychosis for inter-rater reliability (IRR). Subjects rated themselves and were retested within 7 days for test-retest reliability. The Brief Psychiatric Rating Scale (BPRS) was administered for convergent validity and chart diagnosis, and the Structured Clinical Interview (SCID) was used to test psychosis discriminant validity. Results Development and calibration study included 649 psychotic patients. Simulations revealed a correlation of r = .92 with the total 73-item bank score, using an average of 12 items. Validation study included 160 additional patients and 40 healthy controls. CAT-Psychosis showed convergent validity (clinician: r = 0.690; 95% confidence interval [95% CI]: 0.610–0.757; self-report: r = .690; 95% CI: 0.609–0.756), IRR (intraclass correlation coefficient [ICC] = 0.733; 95% CI: 0.611–0.828), and test-retest reliability (clinician ICC = 0.862; 95% CI: 0.767–0.922; self-report ICC = 0.815; 95%CI: 0.741–0.871). CAT-Psychosis could discriminate psychosis from healthy controls (clinician: area under the receiver operating characteristic curve [AUC] = 0.965, 95% CI: 0.945–0.984; self-report AUC = 0.850, 95% CI: 0.807–0.894). The median length of the clinician-administered assessment was 5 minutes (interquartile range [IQR]: 3:23–8:29 min) and 1 minute, 20 seconds (IQR: 0:57–2:09 min) for the self-report. Conclusion CAT-Psychosis can quickly and reliably assess the severity of psychosis and discriminate psychotic patients from healthy controls, creating an opportunity for frequent remote assessment and patient/population-level follow-up.

The Current State and Validity of Digital Assessment Tools for Psychiatry: Systematic Review

“…Most of the studies included participants from primary care services or the general population (18/28, 64% [ 20 , 22 - 25 , 28 , 32 , 35 , 37 - 45 , 47 ]). This was followed by the inclusion of participants from secondary care or specialist services, including psychiatric outpatients (12/28, 43% [ 20 , 27 , 29 - 31 , 33 - 35 , 38 , 45 - 47 ]). Of the 28 studies, 6 (21%) included university students [ 21 , 23 - 26 , 36 ], whereas 4 (14%) purposely recruited nonpsychiatric controls [ 29 - 31 , 33 ].…”

“…Sample sizes ranged from 100 [ 44 ] to 6361 [ 45 ], with all but 3 studies [ 26 , 27 , 33 ] including a larger proportion of women. The mean age across studies ranged from 20 [ 26 ] to 53 years [ 44 ], although not all studies provided this information.…”

“…In total, 1 study (1/28, 4%) targeted psychosis [ 33 ] using the Computerized Adaptive Test–Psychosis (CAT–Psychosis), which is one of the tests available in the CAT–MH. The accuracy of the CAT–Psychosis was good (entire sample: AUC=0.85; including only those who had received the Structured Clinical Interview for DSM Axis I Disorders: AUC=0.80).…”

“…This assessment revealed a high risk of bias in most of the considered studies. For instance, with regard to patient selection, 12 studies (12/28, 43%) [ 20 , 24 , 29 - 33 , 38 , 40 , 42 , 45 , 47 ] had high risk of bias, primarily because of issues with enrollment and a failure to avoid a case–control sample, which may not fully reflect real-world patient populations. A total of 9 studies (9/28, 32%) [ 21 - 23 , 25 , 34 , 36 , 39 , 44 , 46 ] did not provide enough information regarding their sample and sampling procedures.…”

Development and Validation of Computerized Adaptive Assessment Tools for the Measurement of Posttraumatic Stress Disorder Among US Military Veterans

“…Evidence for other mental health conditions (ie, depression, 16 anxiety, 17 mania/hypomania, 18 psychosis, 19 suicide risk, 20 and substance use disorders 21 ) indicates that one can create large item banks (hundreds of items for a given disorder), from which a small optimal subset of items can be adaptively administered for a given individual with no or minimal loss of information, yielding a substantial reduction in patient and clinician burden while maintaining high sensitivity and specificity for diagnostic categorization, as well as high correlation with extant self-and clinician-rated symptom severity standard measures. For CAD, Gibbons et al 12 noted that the CAD for diagnosis of major depressive disorder reproduced the hour-long Structured Clinical Interview for DSM-5 Research Version (SCID) 13 diagnosis of major depressive disorder in less than a minute, using an average of 4 adaptively selected self-reported items, while maintaining sensitivity of 0.95 and specificity of 0.87 for the clinical DSM-5 diagnosis.…”