Neural networks based on attention architecture are robust to data missingness for early predicting hospital mortality in intensive care unit patients

Abstract

Background Although the machine learning model developed on electronic health records has become a promising method for early predicting hospital mortality, few studies focus on the approaches for handling missing data in electronic health records and evaluate model robustness to data missingness. This study proposes an attention architecture that shows excellent predictive performance and is robust to data missingness. Methods Two public intensive care unit databases were used for model training and external validation, respectively. Three neural networks (masked attention model, attention model with imputation, attention model with missing indicator) based on the attention architecture were developed, using masked attention mechanism, multiple imputation, and missing indicator to handle missing data, respectively. Model interpretability was analyzed by attention allocations. Extreme gradient boosting, logistic regression with multiple imputation and missing indicator (logistic regression with imputation, logistic regression with missing indicator) were used as baseline models. Model discrimination and calibration were evaluated by area under the receiver operating characteristic curve, area under precision-recall curve, and calibration curve. In addition, model robustness to data missingness in both model training and validation was evaluated by three analyses. Results In total, 65,623 and 150,753 intensive care unit stays were respectively included in the training set and the test set, with mortality of 10.1% and 8.5%, and overall missing rate of 10.3% and 19.7%. attention model with missing indicator had the highest area under the receiver operating characteristic curve (0.869; 95% CI: 0.865 to 0.873) in external validation; attention model with imputation had the highest area under precision-recall curve (0.497; 95% CI: 0.480–0.513). Masked attention model and attention model with imputation showed better calibration than other models. The three neural networks showed different patterns of attention allocation. In terms of robustness to data missingness, masked attention model and attention model with missing indicator are more robust to missing data in model training; while attention model with imputation is more robust to missing data in model validation. Conclusions The attention architecture has the potential to become an excellent model architecture for clinical prediction task with data missingness.

The Standardized Mortality Ratio and ICU Benchmarking: An Old Measure That Is Still Missing the Mark*

“…Few studies have evaluated the effect of AI and mortality prediction, however, in one study a trained neural network, a machine learning process, used less variables and outperformed mortality predictions using the APACHE II score in critically ill adults (18). Neural networks were shown to be robust predictors of mortality applied to publicly available ICU databases even with missing patient data points (19). Until new models and methodologies are widely used, we are left with little progress forward and the same limitations in models for mortality prediction in the ICU.…”