Validation of an Electronic Health Record–Based Suicide Risk Prediction Modeling Approach Across Multiple Health Care Systems

Abstract

IMPORTANCE Suicide is a leading cause of mortality, with suicide-related deaths increasing in recent years. Automated methods for individualized risk prediction have great potential to address this growing public health threat. To facilitate their adoption, they must first be validated across diverse health care settings.OBJECTIVE To evaluate the generalizability and cross-site performance of a risk prediction method using readily available structured data from electronic health records in predicting incident suicide attempts across multiple, independent, US health care systems. DESIGN, SETTING, AND PARTICIPANTSFor this prognostic study, data were extracted from longitudinal electronic health record data comprising International Classification of Diseases, Ninth Revision diagnoses, laboratory test results, procedures codes, and medications for more than 3.7 million patients from 5 independent health care systems participating in the Accessible Research Commons for Health network. Across sites, 6 to 17 years' worth of data were available, up to 2018. Outcomes were defined by International Classification of Diseases, Ninth Revision codes reflecting incident suicide attempts (with positive predictive value >0.70 according to expert clinician medical record review). Models were trained using naive Bayes classifiers in each of the 5 systems. Models were cross-validated in independent data sets at each site, and performance metrics were calculated. Data analysis was performed from November 2017 to August 2019. MAIN OUTCOMES AND MEASURES The primary outcome was suicide attempt as defined by a previously validated case definition using International Classification of Diseases, Ninth Revision codes. The accuracy and timeliness of the prediction were measured at each site. RESULTS Across the 5 health care systems, of the 3 714 105 patients (2 130 454 female [57.2%])included in the analysis, 39 162 cases (1.1%) were identified. Predictive features varied by site but, as expected, the most common predictors reflected mental health conditions (eg, borderline personality disorder, with odds ratios of 8.1-12.9, and bipolar disorder, with odds ratios of 0.9-9.1) and substance use disorders (eg, drug withdrawal syndrome, with odds ratios of 7.0-12.9). Despite variation in geographical location, demographic characteristics, and population health characteristics, model performance was similar across sites, with areas under the curve ranging from 0.71 (95% CI, 0.70-0.72) to 0.76 (95% CI, 0.75-0.77). Across sites, at a specificity of 90%, the models detected a mean of 38% of cases a mean of 2.1 years in advance. CONCLUSIONS AND RELEVANCEAcross 5 diverse health care systems, a computationally efficient approach leveraging the full spectrum of structured electronic health record data was able to detect (continued) Key Points Question Can a process for training machine-learning algorithms based on electronic health records identify individuals at increased risk of suicide attempts across independent health care systems? Abstract (contin...

Design, Implementation and Feasibility of an Informatics Infrastructure for Standardized Data Acquisition, Transfer, Storage and Export in Psychiatric Clinical Routine

“…With the present study, we extend a previous line of research on predictive modeling based on EHR data. While previous studies have demonstrated empirical evidence for the predictive validity of EHR data in psychiatric use cases [18][19][20], to the best of our knowledge the present study is the first to not only report on the design but also on successful implementation and technical feasibility of the informatics infrastructure for standardized acquisition, transfer, storage and access of real world data for analytic purposes in psychiatric care which is the basic requirement for the application and validation of predictive models in future clinical studies.…”

“…Hence, ecologically valid predictive models would require access to standardized real world data collected at the point-of-care [17]. Importantly, large-scale studies reporting the successful application of multivariate models trained on data from Electronic Health Record (EHR) including features such as diagnosis and procedures, laboratory parameters and medication for the prediction of suicide risk or weight gain following antidepressant treatment have demonstrated the capacity and generalizability of predictive models trained on real-world data [18][19][20]. Further extension of EHRs via standardized collection of predictive variables such as known risk factors might further enhance the potential of this novel data entity for predictive analytics in psychiatry [21,22].…”

Decomposing implicit associations about life and death improves our understanding of suicidal behavior

“…Instead, the IAT could be used in conjunction with additional metrics with good psychometric properties to more comprehensively evaluate risk. For instance, advances in machine learning techniques could be used to help identify the most at‐risk individuals (e.g., Barak‐Corren et al., 2020; Ribeiro, Huang, Fox, Walsh, & Linthicum, 2019).…”

Characterization of time-variant and time-invariant assessment of suicidality on Reddit using C-SSRS

“…The time-variant changes in these factors make the task of diagnosis for MHPs more challenging [ 15 ]. Even though Electronic Health Records (EHRs) are longitudinal, studies have predominantly relied on time-invariant modeling of content to predict suicide-related ideations, suicide-related behaviors, and suicide attempt [ 16 , 17 ]. This approach is often employed due to patients’ low engagement and poor treatment adherence resulting ill-informed follow-up diagnostic procedure.…”