Identifying Rare Circumstances Preceding Female Firearm Suicides: Validating A Large Language Model Approach

Abstract

Background Firearm suicide has been more prevalent among males, but age-adjusted female firearm suicide rates increased by 20% from 2010 to 2020, outpacing the rate increase among males by about 8 percentage points, and female firearm suicide may have different contributing circumstances. In the United States, the National Violent Death Reporting System (NVDRS) is a comprehensive source of data on violent deaths and includes unstructured incident narrative reports from coroners or medical examiners and law enforcement. Conventional natural language processing approaches have been used to identify common circumstances preceding female firearm suicide deaths but failed to identify rarer circumstances due to insufficient training data. Objective This study aimed to leverage a large language model approach to identify infrequent circumstances preceding female firearm suicide in the unstructured coroners or medical examiners and law enforcement narrative reports available in the NVDRS. Methods We used the narrative reports of 1462 female firearm suicide decedents in the NVDRS from 2014 to 2018. The reports were written in English. We coded 9 infrequent circumstances preceding female firearm suicides. We experimented with predicting those circumstances by leveraging a large language model approach in a yes/no question-answer format. We measured the prediction accuracy with F1-score (ranging from 0 to 1). F1-score is the harmonic mean of precision (positive predictive value) and recall (true positive rate or sensitivity). Results Our large language model outperformed a conventional support vector machine–supervised machine learning approach by a wide margin. Compared to the support vector machine model, which had F1-scores less than 0.2 for most infrequent circumstances, our large language model approach achieved an F1-score of over 0.6 for 4 circumstances and 0.8 for 2 circumstances. Conclusions The use of a large language model approach shows promise. Researchers interested in using natural language processing to identify infrequent circumstances in narrative report data may benefit from large language models.

Applications of Large Language Models in the Field of Suicide Prevention: A Scoping Review (Preprint)

“…Applications focused on contextual language understanding were predominantly employed for the purposes of identification, detection, or prediction of suicide risk. Generative applications also focused on prediction-based tasks [44][45][46], but extended to the evaluation of suicide risk [47][48][49], the identification of circumstances preceding suicide [50], information retrieval or question-answering systems [51], and creation of mental health nursing care plans [52]. Among the generative applications, seven out of nine utilized the text-based ChatGPT user interface, while the remaining two employed text-based interfaces for an educational BERT model [51] and a data secure FLAN model [50].…”

“…Generative applications also focused on prediction-based tasks [44][45][46], but extended to the evaluation of suicide risk [47][48][49], the identification of circumstances preceding suicide [50], information retrieval or question-answering systems [51], and creation of mental health nursing care plans [52]. Among the generative applications, seven out of nine utilized the text-based ChatGPT user interface, while the remaining two employed text-based interfaces for an educational BERT model [51] and a data secure FLAN model [50]. The prevalence of GPT use in generative applications stands in contrast to the dominance of the use of the BERT model across LLM-based research to date more generally.…”

“…Of the 31 studies employing social media data, user or poster consent for the use of their data was sought in only one study [43]. Of the 12 studies (28%) that did not use social media data, 3 used data from crisis counselling apps or services [57][58][59], with the remainder a heterogenous collection of data from the National Violent Death Report System [50,60], educational or academic documents [51,61], or participant vignettes [49,52]. One study utilized research participant data gathered during the course of the study [62].…”

“…Improved policy was noted as of clinical relevance [59,82], as there may be a lack of policy safeguarding vulnerable people and the use of LLMs [52]. Cost effectiveness and increasing the quality of data annotation were also noted [47], as was use in public health surveillance, potentially allowing practitioners to track the prevalence of infrequent conditions [50].…”

“…Non-transparent data collection and inference processes currently used by social media platforms were highlighted as a growing area of concern [84]. Hallucinations (the tendency for LLMs to sometimes generate false responses), were identified as having impacts for the safety and reliability of LLM applications [47,50]. Some studies raised concerns that models trained on vast amounts of data containing social biases or prejudices may perpetuate stigma towards suicide or self-harm [44,47].…”