Machine Learning for Violence Risk Assessment Using Dutch Clinical Notes

Abstract

Violence risk assessment in psychiatric institutions enables interventions to avoid violence incidents. Clinical notes written by practitioners and available in electronic health records are valuable resources capturing unique information, but are seldom used to their full potential. We explore conventional and deep machine learning methods to assess violence risk in psychiatric patients using practitioner notes. The performance of our best models is comparable to the currently used questionnaire-based method, with an area under the Receiver Operating Characteristic curve of approximately 0.8. We find that the deep-learning model BERTje performs worse than conventional machine learning methods. We also evaluate our data and our classifiers to understand the performance of our models better. This is particularly important for the applicability of evaluated classifiers to new data, and is also of great interest to practitioners, due to the increased availability of new data in electronic format.

Topic Modeling for Interpretable Text Classification From EHRs

“…If the full topic distribution is taken, the coherence score would be the same for all topic models. Note that, the best predictive performance of the LSI model, based on all words, performs almost on par (with the AUC slightly below 0.8) with the best predictive performance in earlier work (Mosteiro et al, 2021 ), and hence, we recommend considering topic embeddings for future text classification approaches is reasonable.…”

“…The data for this research consists of clinical notes, written in Dutch, by nurses and physicians in the University Medical Center (UMC) Utrecht's psychiatry ward between 2012-08-01 and 2020-03-01 as used in previous studies (Mosteiro et al, 2020 , 2021 ; Rijcken et al, 2021 ). The 834,834 notes available are de-identified for patient privacy using DEDUCE (Menger et al, 2018b ).…”

“…To assess the predictive performance of the topic models, we use both the area under the ROC curve (AUC) (Fawcett, 2006 ) and the area under the Kappa curve (AUK) (Kaymak et al, 2012 ). The AUC is one of the most commonly used scalars for ranking model performance and was used in previous work as well (Mosteiro et al, 2020 ), (Menger et al, 2018a ; Mosteiro et al, 2021 ). The AUC is independent of class priors, but it ignores misclassification costs.…”

“…Instead, automated machine-learning approaches based on existing patient information could overcome the time burden and help make more objective predictions. Various automated text classification approaches utilizing clinical notes in the electronic health record allow for more accurate predictions than the questionnaires (Menger et al, 2018a ; Mosteiro et al, 2021 ). In addition to accurate predictions, clinical providers and other decision-makers in healthcare consider the interpretability of model predictions as a priority for implementation and utilization.…”

Federated learning for violence incident prediction in a simulated cross-institutional psychiatric setting

“…Menger et al (2018) methods to convert texts into vectorial representations, including bag-of-words, TF-IDF, Word2Vec and Doc2Vec. Following previous work (Mosteiro et al, 2021), in this paper we use Doc2Vec (Le & Mikolov, 2014), which generates a fixed-length vector for a piece of text of arbitrary length. In this study's context, a document is the collection of notes of one admission period of a patient.…”