Psychosis Prognosis Predictor: A Continuous and Uncertainty-Aware Prediction of Treatment Outcome in First-Episode Psychosis

Abstract

Importance: Presently, clinicians face challenges in accurately predicting the prognosis of patients with psychosis. Although machine learning models have shown promising potential in individual-level outcome prediction, their practical implementation as tools for real-world clinical practice has been hindered by several limitations. These limitations include difficulties in predicting multiple clinical outcomes, effectively capturing the evolving status of patients over time, and establishing trust in machine predictions. Addressing these shortcomings is crucial for responsibly leveraging machine learning in clinical decision-making for psychosis prognosis. Objective: We propose and evaluate a multi-task recurrent neural network architecture specifically designed to overcome these challenges, enabling trustworthy prediction of psychosis prognosis. By developing a model that can effectively handle multiple clinical outcomes, capture the dynamic nature of patients' status over time, and instill confidence in its predictions, we aim to provide clinicians with a robust tool for making informed and responsible decisions in the treatment of psychosis. Design, setting, and participants: The study sample comprised 446 individuals, aged between 18 and 40 years, diagnosed with first-episode psychosis and participating in the OPTiMiSE study. To predict the likelihood of remission, we selected a range of multimodal baseline variables. These variables included PANSS scores (Positive and Negative Syndrome Scale), diagnostic subclass, duration of untreated psychosis, age at onset, sex, as well as physical health and lifestyle variables. The treatment outcome measures assessed at both week 4 and week 10 encompassed symptomatic remission, clinical global remission, and functional remission. Main outcome and measures: we devised a recurrent neural network architecture incorporating long short-term memory (LSTM) units to facilitate outcome prediction by leveraging multimodal baseline variables and clinical data collected at multiple time points. To account for model uncertainty, we employed a novel fuzzy logic approach to integrate the level of uncertainty into individual predictions. To assess the predictive performance of our model, we employed rigorous evaluation techniques, including repeated 10-fold cross-validation and leave-one-site-out cross-validation. Results: When solely utilizing pre-treatment patient status to forecast various outcomes in 4 weeks after starting the treatment, the leave-one-site-out validation process yielded area under the receiver operating characteristic curve (AUC) values ranging from 0.62 to 0.67. Notably, the predictive performance witnessed ~0.04 improvement upon the inclusion of 1-week follow-up patient status, resulting in AUC values ranging from 0.66 to 0.72. Regarding the prediction of outcomes 10 weeks after the start of treatment, the models constructed solely with pre-treatment patient status achieved AUC values between 0.56 and 0.64. However, by incorporating follow-up patient status (specifically after 1, 4, and 6 weeks), the performance of the models was enhanced, resulting in AUC values of 0.72 to 0.74. Furthermore, after incorporating prediction uncertainties and stratifying the model decisions based on model confidence, we could achieve accuracies above 0.8 for ~50% of patients in five out of six clinical scenarios. Conclusions and Relevance: Our approach involved constructing prediction models utilizing a flexible neural network architecture tailored to clinical scenarios derived from a time series dataset. One crucial aspect we incorporated was the consideration of uncertainty in individual predictions, which enhances the reliability of decision-making based on the model's output. Through our study, we provided compelling evidence showcasing the significance of leveraging time series data for achieving more accurate treatment outcome prediction in the field of psychiatry. Therefore, we advocate for the inclusion of time series data in future individualized prediction modeling endeavors within psychiatric research, as it holds substantial promise in improving prognostic accuracy and informing personalized treatment approaches.