Risk prediction of delirium in hospitalized patients using machine learning: An implementation and prospective evaluation study

Abstract

Objective Machine learning models trained on electronic health records have achieved high prognostic accuracy in test datasets, but little is known about their embedding into clinical workflows. We implemented a random forest–based algorithm to identify hospitalized patients at high risk for delirium, and evaluated its performance in a clinical setting. Materials and Methods Delirium was predicted at admission and recalculated on the evening of admission. The defined prediction outcome was a delirium coded for the recent hospital stay. During 7 months of prospective evaluation, 5530 predictions were analyzed. In addition, 119 predictions for internal medicine patients were compared with ratings of clinical experts in a blinded and nonblinded setting. Results During clinical application, the algorithm achieved a sensitivity of 74.1% and a specificity of 82.2%. Discrimination on prospective data (area under the receiver-operating characteristic curve = 0.86) was as good as in the test dataset, but calibration was poor. The predictions correlated strongly with delirium risk perceived by experts in the blinded (r = 0.81) and nonblinded (r = 0.62) settings. A major advantage of our setting was the timely prediction without additional data entry. Discussion The implemented machine learning algorithm achieved a stable performance predicting delirium in high agreement with expert ratings, but improvement of calibration is needed. Future research should evaluate the acceptance of implemented machine learning algorithms by health professionals. Conclusions Our study provides new insights into the implementation process of a machine learning algorithm into a clinical workflow and demonstrates its predictive power for delirium.

Technology Acceptance of a Machine Learning Algorithm Predicting Delirium in a Clinical Setting: a Mixed-Methods Study

“…For every patient admitted to one of the departments, a random forest-based algorithm automatically predicts the delirium risk based on existing EHR data [ 17 ]. The predicted outcome is an ICD-10-GM (International Classification of Diseases – Tenth Revision – German Modification) coded diagnosis F05 (Delirium due to known physiological condition) or mentions of delirium in the text of a patient’s discharge summaries.…”

“…We recently implemented an ML-based application predicting the occurrence of delirium in an Austrian hospital, and prospectively evaluated its performance in a routine clinical setting [ 17 ]. Delirium is a syndrome of acute confusional state with an acute decline of cognitive functioning [ 18 ].…”

“…Identifying patients with highest risk is especially beneficial for nursing, because delirium can be prevented by non-pharmacological interventions [ 20 , 21 ]. During a pilot study of seven months, the performance of the algorithm had achieved a specificity of 82% and a sensitivity of 74% [ 17 ].…”

Evaluation of crowdsourced mortality prediction models as a framework for assessing AI in medicine

“…We previously piloted this method on an EHR dataset from the University of Washington and demonstrated the feasibility of accurate model development without the model developer having direct access to the patient data 8 . This approach has two benefits: (1) it protects patient data while allowing researchers to build machine learning methods and (2) it forces a more standardized and transferable approach to building models allowing the data host to perform rigorous evaluations of submitted models.…”

Advancing care for acute gastrointestinal bleeding using artificial intelligence

“…Machine learning risk prediction tools developed using electronic health record data have shown promise when deployed to predict sepsis, acute kidney injury, and in‐hospital delirium 29–32 . However, there has been limited work in acute GIB.…”