Integration of Face-to-Face Screening With Real-time Machine Learning to Predict Risk of Suicide Among Adults

Abstract

IMPORTANCE Understanding the differences and potential synergies between traditional clinician assessment and automated machine learning might enable more accurate and useful suicide risk detection. OBJECTIVETo evaluate the respective and combined abilities of a real-time machine learning model and the Columbia Suicide Severity Rating Scale (C-SSRS) to predict suicide attempt (SA) and suicidal ideation (SI). DESIGN, SETTING, AND PARTICIPANTS This cohort study included encounters with adult patients (aged Ն18 years) at a major academic medical center. The C-SSRS was administered during routine care, and a Vanderbilt Suicide Attempt and Ideation Likelihood (VSAIL) prediction was generated in the electronic health record. Encounters took place in the inpatient, ambulatory surgical, and emergency department settings. Data were collected from June 2019 to September 2020. MAIN OUTCOMES AND MEASURES Primary outcomes were the incidence of SA and SI, encoded as International Classification of Diseases codes, occurring within various time periods after an index visit. We evaluated the retrospective validity of the C-SSRS, VSAIL, and ensemble models combining both. Discrimination metrics included area under the receiver operating curve (AUROC), area under the precision-recall curve (AUPR), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). RESULTS The cohort included 120 398 unique index visits for 83 394 patients (mean [SD] age, 51.2 [20.6] years; 38 107 [46%] men; 45 273 [54%] women; 13 644 [16%] Black; 63 869 [77%] White).Within 30 days of an index visit, the combined models had higher AUROC (SA: 0.874-0.887; SI: 0.869-0.879) than both the VSAIL (SA: 0.729; SI: 0.773) and C-SSRS (SA: 0.823; SI: 0.777) models.

Randomized Controlled Comparative Effectiveness Trial of Risk Model-Guided Clinical Decision Support for Suicide Screening

“…This pragmatic, two-arm RCT uses a validated risk model to prompt suicide preventive CDS at the start of routine healthcare encounters. 24,32,33,39 Study Setting A non-behavioral health setting with increased suicide risk 40 and variable suicide prevention workflows, ambulatory Neurology clinics serve as the trial setting. Unlike high-risk settings such as the Emergency Department, ambulatory Neurology clinics do not have universal screening protocols in all sites.…”

“…[18][19][20][21][22][23] Recent research suggests that statistical modeling combined with face-to-face screening outperform either alone. 24 To enable prevention, predictive models must be actualized through tools like CDS. Prior literature outside suicide research has examined forms of CDS such as interruptive (e.g., alerts) and non-interruptive (e.g., static icons or visual cues) to inform contact isolation decisions, 25 laboratory alerts, 26 and blood transfusion.…”

“…30,31 Our team has previously validated, replicated, and prospectively "silently" tested an Electronic Health Record (EHR)-based suicide risk model. 24,32,33 Here, we report design and evaluation of CDS driven by that model to prompt suicide risk assessment within healthcare encounters in settings that do not conduct universal screening. The designs of the CDS and the research protocol are informed by Human-Centered Design (HCD), 34 a framework to evaluate appropriate CDS alerts and responses, 35 and a deployment framework for clinical artificial intelligence.…”

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

“…Focussing on suicide risk, Wilimitis et al [51] evaluated automated detection in clinical settings by combining predictions from the Columbia Suicide Severity Rating Scale with a real-time ML model. Combined models outperformed the model alone for risks of suicide attempt and suicidal ideation in a cohort study of 120,398 adult patient encounters in the US.…”

Suicide Risk Prediction in Clinical Settings—Additional Considerations for Face-to-Face Screening and Machine Learning Approaches

“…Elsewhere in JAMA Network Open, Wilimitis and colleagues 1 set out to resolve a central debate in suicide prevention: do face-to-face risk screenings or electronic health record-based machine learning algorithms provide greater public health benefit? They approach this question by proposing and testing a third option, combining the use of these 2 modes.…”