Making Sense of Violence Risk Predictions Using Clinical Notes

Abstract

Violence risk assessment in psychiatric institutions enables interventions to avoid violence incidents. Clinical notes written by practitioners and available in electronic health records (EHR) are valuable resources that are seldom used to their full potential. Previous studies have attempted to assess violence risk in psychiatric patients using such notes, with acceptable performance. However, they do not explain why classification works and how it can be improved. We explore two methods to better understand the quality of a classifier in the context of clinical note analysis: random forests using topic models, and choice of evaluation metric. These methods allow us to understand both our data and our methodology more profoundly, setting up the groundwork to work on improved models that build upon this understanding. This is particularly important when it comes to the generalizability of evaluated classifiers to new data, a trustworthiness problem that is of great interest due to the increased availability of new data in electronic format.

Machine Learning for Violence Risk Assessment Using Dutch Clinical Notes

“…Menger et al [12] use Dutch clinical text to predict violence incidents from patients in treatment facilities. In Mosteiro et al [13] we compared several classical machine learning methods for VRA of EHR notes, including Latent Dirichlet Allocation (LDA) for topic modeling, and we discussed the agreement between some of those classifiers. In this work we extend on that approach, introducing the BVC as a baseline and employing BERT for document classification.…”

“…Nevertheless, violence prediction based on Dutch written notes appears to be a challenging endeavor as no efforts have shown satisfying results, with the AUC stagnating below 0.8 [11][12][13]. In those papers, various machine learning methods were applied, including bag-of-words, document embeddings, and topic modeling to generate numerical representations of texts; and support vector machines (SVM) and random forests for classification.…”

Topic Modeling for Interpretable Text Classification From EHRs

“…The data for this research consists of clinical notes, written in Dutch, by nurses and physicians in the University Medical Center (UMC) Utrecht's psychiatry ward between 2012-08-01 and 2020-03-01 as used in previous studies (Mosteiro et al, 2020 , 2021 ; Rijcken et al, 2021 ). The 834,834 notes available are de-identified for patient privacy using DEDUCE (Menger et al, 2018b ).…”

Machine Learning for Violence Risk Assessment Using Dutch Clinical Notes

“…Menger et al [12] use Dutch clinical text to predict violence incidents from patients in treatment facilities. In [13] we compared several classical machine learning methods for VRA of EHR notes, including Latent Dirichlet Allocation (LDA) for topic modeling, and we discussed the agreement between some of those classifiers. In this work we extend on that approach, introducing the BVC as a baseline and employing BERT for document classification.…”

“…Nevertheless, violence prediction based on Dutch written notes appears to be a challenging endeavour as no efforts have shown satisfying results, with the AUC stagnating below 0.8 [11,12,13]. In those papers, various machine learning methods were applied, including bag-of-words, document embeddings and topic modeling to generate numerical representations of texts; and support vector machines and random forests for classification.…”