Dynamic prediction of psychological treatment outcomes: development and validation of a prediction model using routinely collected symptom data

Abstract

Background Common mental disorders can be effectively treated with psychotherapy, but some patients do not respond well and require timely identification to prevent treatment failure. We aimed to develop and validate a dynamic model to predict psychological treatment outcomes, and to compare the model with currently used methods, including expected treatment response models and machine learning models. MethodsIn this prediction model development and validation study, we obtained data from two UK studies including patients who had accessed therapy via Improving Access to Psychological Therapies (IAPT) services managed by ten UK National Health Service (NHS) Trusts between March, 2012, and June, 2018, to predict treatment outcomes. In study 1, we used data on patient-reported depression (Patient Health Questionnaire 9 [PHQ-9]) and anxiety (Generalised Anxiety Disorder 7 [GAD-7]) symptom measures obtained on a session-by-session basis (Leeds Community Healthcare NHS Trust dataset; n=2317) to train the Oracle dynamic prediction model using iterative logistic regression analysis. The outcome of interest was reliable and clinically significant improvement in depression (PHQ-9) and anxiety (GAD-7) symptoms. The predictive accuracy of the model was assessed in an external test sample (Cumbria Northumberland Tyne and Wear NHS Foundation Trust dataset; n=2036) using the area under the curve (AUC), positive predictive values (PPVs), and negative predictive values (NPVs). In study 2, we retrained the Oracle algorithm using a multiservice sample (South West Yorkshire Partnership NHS Foundation Trust, North East London NHS Foundation Trust, Cheshire and Wirral Partnership NHS Foundation Trust, and Cambridgeshire and Peterborough NHS Foundation Trust; n=42 992) and compared its performance with an expected treatment response model and five machine learning models (Bayesian updating algorithm, elastic net regularisation, extreme gradient boosting, support vector machine, and neural networks based on a multilayer perceptron algorithm) in an external test sample (Whittington Health NHS Trust; Barnet Enfield and Haringey Mental Health Trust; Pennine Care NHS Foundation Trust; and Humber NHS Foundation Trust; n=30 026). Findings The Oracle algorithm trained using iterative logistic regressions generalised well to external test samples, explaining up to 47•3% of variability in treatment outcomes. Prediction accuracy was modest at session one (AUC 0•59 [95% CI 0•55-0•62], PPV 0•63, NPV 0•61), but improved over time, reaching high prediction accuracy (AUC 0•81 [0•77-0•86], PPV 0•79, NPV 0•69) as early as session seven. The performance of the Oracle model was similar to complex (eg, including patient profiling variables) and computationally intensive machine learning models (eg, neural networks based on a multilayer perceptron algorithm, extreme gradient boosting). Furthermore, the predictive accuracy of a more simple dynamic algorithm including only baseline and index-session scores was comparable to more complex algorithms that includ...

Predicting relapse or recurrence of depression: systematic review of prognostic models

“…7 There have been an increasing number of attempts to derive and validate prognostic models to predict depression-related outcomes. [8][9][10][11] There has been no previous systematic review to identify all prognostic models designed to predict relapse or recurrence of depression.…”

Uncovering the linguistic characteristics of psychotherapy: a computational approach to measure therapist language timing, responsiveness, and consistency

“…If successful, computational language analysis of entire psychotherapy sessions may address long-standing criticisms of methodological rigor in psychotherapy evaluation 15,72 . If deployed ethically and fairly, this approach could assist evaluations of treatment adherence and quality 15,[73][74][75][76][77] . To appreciate the full diversity of expression in therapy, computationally-conducted, theoretically informed evaluation may be a practical necessity 14,78 .…”

Personalized prediction of response to smartphone-delivered meditation training

“…Other potentially fruitful future directions include evaluating a broader set of patient characteristics previously shown or hypothesized to predict likelihood of response to different interventions (Kessler et al, 2017). In addition, prediction models could be developed using data drawn from large naturalistic datasets evaluating mHealth interventions, as has been done for inperson psychotherapy and pharmacotherapy (Bone et al, 2021;Webb et al, 2020;Webb, Forgeard, et al, 2021). In addition to testing the utility of these models in "realworld" settings, naturalistic settings often provide large datasets relative to RCTs and thus can increase statistical power (Luedtke et al, 2019).…”