Real-Time Clinical Decision Support Based on Recurrent Neural Networks for In-Hospital Acute Kidney Injury: External Validation and Model Interpretation

Abstract

Background Acute kidney injury (AKI) is commonly encountered in clinical practice and is associated with poor patient outcomes and increased health care costs. Despite it posing significant challenges for clinicians, effective measures for AKI prediction and prevention are lacking. Previously published AKI prediction models mostly have a simple design without external validation. Furthermore, little is known about the process of linking model output and clinical decisions due to the black-box nature of neural network models. Objective We aimed to present an externally validated recurrent neural network (RNN)–based continuous prediction model for in-hospital AKI and show applicable model interpretations in relation to clinical decision support. Methods Study populations were all patients aged 18 years or older who were hospitalized for more than 48 hours between 2013 and 2017 in 2 tertiary hospitals in Korea (Seoul National University Bundang Hospital and Seoul National University Hospital). All demographic data, laboratory values, vital signs, and clinical conditions of patients were obtained from electronic health records of each hospital. We developed 2-stage hierarchical prediction models (model 1 and model 2) using RNN algorithms. The outcome variable for model 1 was the occurrence of AKI within 7 days from the present. Model 2 predicted the future trajectory of creatinine values up to 72 hours. The performance of each developed model was evaluated using the internal and external validation data sets. For the explainability of our models, different model-agnostic interpretation methods were used, including Shapley Additive Explanations, partial dependence plots, individual conditional expectation, and accumulated local effects plots. Results We included 69,081 patients in the training, 7675 in the internal validation, and 72,352 in the external validation cohorts for model development after excluding cases with missing data and those with an estimated glomerular filtration rate less than 15 mL/min/1.73 m2 or end-stage kidney disease. Model 1 predicted any AKI development with an area under the receiver operating characteristic curve (AUC) of 0.88 (internal validation) and 0.84 (external validation), and stage 2 or higher AKI development with an AUC of 0.93 (internal validation) and 0.90 (external validation). Model 2 predicted the future creatinine values within 3 days with mean-squared errors of 0.04-0.09 for patients with higher risks of AKI and 0.03-0.08 for those with lower risks. Based on the developed models, we showed AKI probability according to feature values in total patients and each individual with partial dependence, accumulated local effects, and individual conditional expectation plots. We also estimated the effects of feature modifications such as nephrotoxic drug discontinuation on future creatinine levels. Conclusions We developed and externally validated a continuous AKI prediction model using RNN algorithms. Our model could provide real-time assessment of future AKI occurrences and individualized risk factors for AKI in general inpatient cohorts; thus, we suggest approaches to support clinical decisions based on prediction models for in-hospital AKI.

Machine learning for early discrimination between transient and persistent acute kidney injury in critically ill patients with sepsis

“…Due to the complexity of SA-AKI, the clinical model integrating routine parameters may be more effective for predicting short-term reversibility of AKI than any parameter considered alone. A possible way to achieve this is to utilize advanced machine learning approaches, which have been applied in the prevention and management of AKI, such as predicting the development of AKI 37 – 41 , volume responsiveness in patients with oliguria 42 and mortality in critically ill AKI patients 43 – 45 . Our study corroborated the promise indicated by these previous studies and extended them by demonstrating the applicability of machine learning methods for predicting persistent AKI in a large cohort of SA-AKI patients.…”

“…Le et al proposed a convolutional neural networks prediction system, which outperformed the XGB model and the SOFA score in predicting AKI 48 h before onset in ICU patients 40 . Similarly, Kim et al used recurrent neural network to assess future AKI occurrence and individualized AKI risk factors in real time among hospitalized patients 41 . Hofer et al applied the deep neural networks to create models for postoperative AKI, mortality, reintubation, and the combined outcome, which exhibited superior performance to the ASA score 46 .…”

Systematic Review of Supervised Machine Learning Models in Prediction of Medical Conditions

“…Two RNN algorithms were created using a dataset of more than 72,000 patients. [72] Model 1 predicted the occurrence of AKI within 7 days with AUROC of 0.84 and model 2 predicted the future trajectory of creatinine values up to 72 hours with AUROC of 0.9. Further development of the suggested approaches could incorporate the model into CDS systems for prediction of in-hospital AKI.…”

“…RNN models capture the temporal nature of EHR, imaging and other medical data to predict diseases, complications, and outcomes. [66][67][68][69][70][71][72][73] Deep learning models produce higher accuracy but suffer from issues of interpretability and instability. [15,75] Combinations of traditional and deep learning models could address challenges of interpretability and accuracy.…”

“…Two RNN algorithms were created using a dataset of more than 72,000 patients. [72] Model 1 predicted the occurrence of AKI within 7 days with AUROC of 0.84 and model 2 predicted the future trajectory of creatinine values up to 72 hours with AUROC of 0.9.…”