Delirium prediction in the ICU: designing a screening tool for preventive interventions

Abstract

Introduction Delirium occurrence is common and preventive strategies are resource intensive. Screening tools can prioritize patients at risk. Using machine learning, we can capture time and treatment effects that pose a challenge to delirium prediction. We aim to develop a delirium prediction model that can be used as a screening tool. Methods From the eICU Collaborative Research Database (eICU-CRD) and the Medical Information Mart for Intensive Care version III (MIMIC-III) database, patients with one or more Confusion Assessment Method-Intensive Care Unit (CAM-ICU) values and intensive care unit (ICU) length of stay greater than 24 h were included in our study. We validated our model using 21 quantitative clinical parameters and assessed performance across a range of observation and prediction windows, using different thresholds and applied interpretation techniques. We evaluate our models based on stratified repeated cross-validation using 3 algorithms, namely Logistic Regression, Random Forest, and Bidirectional Long Short-Term Memory (BiLSTM). BiLSTM represents an evolution from recurrent neural network-based Long Short-Term Memory, and with a backward input, preserves information from both past and future. Model performance is measured using Area Under Receiver Operating Characteristic, Area Under Precision Recall Curve, Recall, Precision (Positive Predictive Value), and Negative Predictive Value metrics. Results We evaluated our results on 16 546 patients (47% female) and 6294 patients (44% female) from eICU-CRD and MIMIC-III databases, respectively. Performance was best in BiLSTM models where, precision and recall changed from 37.52% (95% confidence interval [CI], 36.00%–39.05%) to 17.45 (95% CI, 15.83%–19.08%) and 86.1% (95% CI, 82.49%–89.71%) to 75.58% (95% CI, 68.33%–82.83%), respectively as prediction window increased from 12 to 96 h. After optimizing for higher recall, precision and recall changed from 26.96% (95% CI, 24.99%–28.94%) to 11.34% (95% CI, 10.71%–11.98%) and 93.73% (95% CI, 93.1%–94.37%) to 92.57% (95% CI, 88.19%–96.95%), respectively. Comparable results were obtained in the MIMIC-III cohort. Conclusions Our model performed comparably to contemporary models using fewer variables. Using techniques like sliding windows, modification of threshold to augment recall and feature ranking for interpretability, we addressed shortcomings of current models.

New onset delirium prediction using machine learning and long short-term memory (LSTM) in electronic health record

“…We identified a recent study that developed an LSTM-based model to predict delirium status at least 24 h after hospitalization based on 21 features. 49 Our study used a more extensive set of over 900 features, while the machine learning part we introduced in the combined model could provide predictions when there was not enough historical data to run the LSTM-based model. In addition, as a critical step for implementation in the clinic, the performance of the prediction model should be considered.…”

TRAPOD: A Transformer Architecture Exploits Intraoperative Temporal Dynamics Improving the Prediction of Postoperative Delirium

“…Respective models have been rarely implemented into clinical practice, though [15][16][17] . Most of the previous studies used logistic regression (LR) models with predisposing risk factors as inputs and CAM-ICU scores as labels to be predicted 15,16,[18][19][20][21][22][23][24] . Temporal signals in feature time series were commonly aggregated with summary statistics arranged into tabular datasets 15 .…”

“…Multiple studies used non-linear machine learning (ML) algorithms, primarily tree-based (TB) models, trained on more complex clinical feature sets 18,19,[27][28][29][30] . Bishara et al 29 developed a prognostic XGBoost 31 TB model with 85.1% AUROC and a precision of 14.4% at 80.6% recall 32 .…”

Evaluating the performance of machine learning methods for risk estimation of delirium in patients hospitalized from the emergency department

“…[10][11][12] There is a need to identify an optimal screening strategy for delirium beyond cognitive assessment because, until we have it, delirium will likely remain an elusive diagnosis. In recent years, numerous studies have developed models in the hospital setting for estimating the risk of delirium in postoperative patients [13][14][15][16][17][18][19][20][21][22] and ICU patients, [23][24][25][26][27][28] but limited work has focused on the ED patient population. 3,29,30 Our objective was to leverage electronic health records to identify a clinically valuable risk estimation model for positive delirium screens in patients admitted from the ED to inpatient units.…”

“…There is a need to identify an optimal screening strategy for delirium beyond cognitive assessment because, until we have it, delirium will likely remain an elusive diagnosis. In recent years, numerous studies have developed models in the hospital setting for estimating the risk of delirium in postoperative patients 13–22 and ICU patients, 23–28 but limited work has focused on the ED patient population 3,29,30 …”