Background This study evaluated whether natural language processing (NLP) of psychotherapy note text provides additional accuracy over and above currently used suicide prediction models. Methods We used a cohort of Veterans Health Administration (VHA) users diagnosed with post-traumatic stress disorder (PTSD) between 2004–2013. Using a case-control design, cases (those that died by suicide during the year following diagnosis) were matched to controls (those that remained alive). After selecting conditional matches based on having shared mental health providers, we chose controls using a 5:1 nearest-neighbor propensity match based on the VHA's structured Electronic Medical Records (EMR)-based suicide prediction model. For cases, psychotherapist notes were collected from diagnosis until death. For controls, psychotherapist notes were collected from diagnosis until matched case's date of death. After ensuring similar numbers of notes, the final sample included 246 cases and 986 controls. Notes were analyzed using Sentiment Analysis and Cognition Engine, a Python-based NLP package. The output was evaluated using machine-learning algorithms. The area under the curve (AUC) was calculated to determine models' predictive accuracy. Results NLP derived variables offered small but significant predictive improvement (AUC = 0.58) for patients that had longer treatment duration. A small sample size limited predictive accuracy. Conclusions Study identifies a novel method for measuring suicide risk over time and potentially categorizing patient subgroups with distinct risk sensitivities. Findings suggest leveraging NLP derived variables from psychotherapy notes offers an additional predictive value over and above the VHA's state-of-the-art structured EMR-based suicide prediction model. Replication with a larger non-PTSD specific sample is required.

Rationale aims and objectives As quality measurement becomes increasingly reliant on the availability of structured electronic medical record (EMR) data, clinicians are asked to perform documentation using tools that facilitate data capture. These tools may not be available, feasible, or acceptable in all clinical scenarios. Alternative methods of assessment, including natural language processing (NLP) of clinical notes, may improve the completeness of quality measurement in real‐world practice. Our objective was to measure the quality of care for a set of evidence‐based practices using structured EMR data alone, and then supplement those measures with additional data derived from NLP. Method As a case example, we studied the quality of care for posttraumatic stress disorder (PTSD) in the United States Department of Veterans Affairs (VA) over a 20‐year period. We measured two aspects of PTSD care, including delivery of evidence‐based psychotherapy (EBP) and associated use of measurement‐based care (MBC), using structured EMR data. We then recalculated these measures using additional data derived from NLP of clinical note text. Results There were 2 098 389 VA patients with a diagnosis of PTSD between 2000 and 2019, 72% (n = 1 515 345) of whom had not previously received EBP for PTSD and were treated after a 2015 mandate to document EBP using templates that generate structured EMR data. Using structured EMR data, we determined that 3.2% (n = 48 004) of those patients met our EBP for PTSD quality standard between 2015 and 2019, and 48.1% (n = 23 088) received associated MBC. With the addition of NLP‐derived data, estimates increased to 4.1% (n = 62 789) and 58.0% (n = 36 435), respectively. Conclusion Healthcare quality data can be significantly improved by supplementing structured EMR data with NLP‐derived data. By using NLP, health systems may be able to fill the gaps in documentation when structured tools are not yet available or there are barriers to using them in clinical practice.

No abstract