Improving ascertainment of suicidal ideation and suicide attempt with natural language processing

Bejan, Cosmin Adrian; Ripperger, Michael; Wilimitis, Drew; Ahmed, Ryan; Kang, Jooeun; Robinson, Katelyn; Morley, Theodore; Ruderfer, Douglas M.; Walsh, Colin G.

doi:10.1038/s41598-022-19358-3

Cited by 16 publications

(12 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Diagnostic codes have been shown to have high 5 s positive predictive value (PPV) in International Classification of Diseases, version 10, in our prior research including PPV of 0.85 for suicide attempt and 0.96 for suicidal ideation. 44 We ascertain hospitalization or emergency department utilization with EHR healthcare encounter data.…”

Section: Secondary Outcomesmentioning

confidence: 99%

Randomized Controlled Comparative Effectiveness Trial of Risk Model-Guided Clinical Decision Support for Suicide Screening

Walsh,

Ripperger,

Novak

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Suicide prevention requires risk identification, appropriate intervention, and follow-up. Traditional risk identification relies on patient self-reporting, support network reporting, or face-to-face screening with validated instruments or history and physical exam. In the last decade, statistical risk models have been studied and more recently deployed to augment clinical judgment. Models have generally been found to be low precision or problematic at scale due to low incidence. Few have been tested in clinical practice, and none have been tested in clinical trials to our knowledge.MethodsWe report the results of a pragmatic randomized controlled trial (RCT) in three outpatient adult Neurology clinic settings. This two-arm trial compared the effectiveness of Interruptive and Non-Interruptive Clinical Decision Support (CDS) to prompt further screening of suicidal ideation for those predicted to be high risk using a real-time, validated statistical risk model of suicide attempt risk, with the decision to screen as the primary end point. Secondary outcomes included rates of suicidal ideation and attempts in both arms. Manual chart review of every trial encounter was used to determine if suicide risk assessment was subsequently documented.ResultsFrom August 16, 2022, through February 16, 2023, our study randomized 596 patient encounters across 561 patients for providers to receive either Interruptive or Non-Interruptive CDS in a 1:1 ratio. Adjusting for provider cluster effects, Interruptive CDS led to significantly higher numbers of decisions to screen (42%=121/289 encounters) compared to Non-Interruptive CDS (4%=12/307) (odds ratio=17.7, p-value <0.001). Secondarily, no documented episodes of suicidal ideation or attempts occurred in either arm. While the proportion of documented assessments among those noting the decision to screen was higher for providers in the Non-Interruptive arm (92%=11/12) than in the Interruptive arm (52%=63/121), the interruptive CDS was associated with more frequent documentation of suicide risk assessment (63/289 encounters compared to 11/307, p-value<0.001).ConclusionsIn this pragmatic RCT of real-time predictive CDS to guide suicide risk assessment, Interruptive CDS led to higher numbers of decisions to screen and documented suicide risk assessments. Well-powered large-scale trials randomizing this type of CDS compared to standard of care are indicated to measure effectiveness in reducing suicidal self-harm.ClinicalTrials.govIdentifier:NCT05312437

show abstract

Section: Secondary Outcomesmentioning

confidence: 99%

Randomized Controlled Comparative Effectiveness Trial of Risk Model-Guided Clinical Decision Support for Suicide Screening

Walsh,

Ripperger,

Novak

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In recent years, there has been a growing interest in using NLP techniques for suicide prevention [30], [31]. Researchers have developed suicide detection systems to analyze and interpret social media data, including text data.…”

Section: B Suicidal Ideation Detection Using Nlpmentioning

confidence: 99%

Socially Aware Synthetic Data Generation for Suicidal Ideation Detection Using Large Language Models

Ghanadian,

Nejadgholi,

Osman

2024

IEEE Access

View full text Add to dashboard Cite

Suicidal ideation detection is a vital research area that holds great potential for improving mental health support systems. However, the sensitivity surrounding suicide-related data poses challenges in accessing large-scale, annotated datasets necessary for training effective machine learning models. To address this limitation, we introduce an innovative strategy that leverages the capabilities of generative AI models, such as ChatGPT, Flan-T5, and Llama, to create synthetic data for suicidal ideation detection. Our data generation approach is grounded in social factors extracted from psychology literature and aims to ensure coverage of essential information related to suicidal ideation. In our study, we benchmarked against state-of-the-art NLP classification models, specifically, those centered around the BERT family structures. When trained on the real-world dataset, UMD, these conventional models tend to yield F1-scores ranging from 0.75 to 0.87. Our synthetic data-driven method, informed by social factors, offers consistent F1scores of 0.82 for both models, suggesting that the richness of topics in synthetic data can bridge the performance gap across different model complexities. Most impressively, when we combined a mere 30% of the UMD dataset with our synthetic data, we witnessed a substantial increase in performance, achieving an F1-score of 0.88 on the UMD test set. Such results underscore the cost-effectiveness and potential of our approach in confronting major challenges in the field, such as data scarcity and the quest for diversity in data representation.

show abstract

“…Our team has published extensively in the suicide informatics literature on developing, validating, and deploying scalable predictive models of suicide risk into practice. As a result, suicide attempt was defined based on prior work using published diagnostic code sets for the 7 silver standard 27 and domain knowledge-driven definitions for the gold standard annotation (see below for details on both).…”

Section: Phenotype Definitionsmentioning

confidence: 99%

“…23 In prior work, we applied NLP to ascertain evidence of any suicide attempt or record of suicidal ideation from EHRs across entire healthcare records. 27 In this work, the intent was to ascertain evidence of new, clinically distinct incidents of these phenotypes. The team discussed numerous potential temporal windows to focus the NLP including: i) healthcare visit episodes; ii) set time-windows, e.g., twenty-four-hour periods; iii) combinations of those two, e.g., clinical episodes plus/minus a time-window to capture documentation lag.…”

Section: Temporalitymentioning

confidence: 99%

“…25,26 In work motivating this study, NLP has been applied to suicidal ideation and suicide attempt yielding accurate and precise ascertainment from unstructured text data agnostic to source data or EHR. 27 To improve scalable incident detection using unstructured healthcare data, we developed and validated a novel incident phenotyping approach using unstructured clinical textual data agnostic to EHR and note type. It's based on a published, validated approach used to ascertain social determinants of health and suicidality across entire healthcare records (prevalence).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Scalable Incident Detection via Natural Language Processing and Probabilistic Language Models

Walsh,

Wilimitis,

Chen

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Post marketing safety surveillance depends in part on the ability to detect concerning clinical events at scale. Spontaneous reporting might be an effective component of safety surveillance, but it requires awareness and understanding among healthcare professionals to achieve its potential. Reliance on readily available structured data such as diagnostic codes risk under-coding and imprecision. Clinical textual data might bridge these gaps, and natural language processing (NLP) has been shown to aid in scalable phenotyping across healthcare records in multiple clinical domains. In this study, we developed and validated a novel incident phenotyping approach using unstructured clinical textual data agnostic to Electronic Health Record (EHR) and note type. It’s based on a published, validated approach (PheRe) used to ascertain social determinants of health and suicidality across entire healthcare records. To demonstrate generalizability, we validated this approach on two separate phenotypes that share common challenges with respect to accurate ascertainment: 1) suicide attempt; 2) sleep-related behaviors. With samples of 89,428 records and 35,863 records for suicide attempt and sleep-related behaviors, respectively, we conducted silver standard (diagnostic coding) and gold standard (manual chart review) validation. We showed Area Under the Precision-Recall Curve of ∼ 0.77 (95% CI 0.75-0.78) for suicide attempt and AUPR ∼ 0.31 (95% CI 0.28-0.34) for sleep-related behaviors. We also evaluated performance by coded race and demonstrated differences in performance by race were dissimilar across phenotypes and require algorithmovigilance and debiasing prior to implementation.

show abstract

Improving ascertainment of suicidal ideation and suicide attempt with natural language processing

Cited by 16 publications

References 30 publications

Randomized Controlled Comparative Effectiveness Trial of Risk Model-Guided Clinical Decision Support for Suicide Screening

Randomized Controlled Comparative Effectiveness Trial of Risk Model-Guided Clinical Decision Support for Suicide Screening

Socially Aware Synthetic Data Generation for Suicidal Ideation Detection Using Large Language Models

Scalable Incident Detection via Natural Language Processing and Probabilistic Language Models

Contact Info

Product

Resources

About