Predicting Postoperative Mortality With Deep Neural Networks and Natural Language Processing: Model Development and Validation

Abstract

Background Machine learning (ML) achieves better predictions of postoperative mortality than previous prediction tools. Free-text descriptions of the preoperative diagnosis and the planned procedure are available preoperatively. Because reading these descriptions helps anesthesiologists evaluate the risk of the surgery, we hypothesized that deep learning (DL) models with unstructured text could improve postoperative mortality prediction. However, it is challenging to extract meaningful concept embeddings from this unstructured clinical text. Objective This study aims to develop a fusion DL model containing structured and unstructured features to predict the in-hospital 30-day postoperative mortality before surgery. ML models for predicting postoperative mortality using preoperative data with or without free clinical text were assessed. Methods We retrospectively collected preoperative anesthesia assessments, surgical information, and discharge summaries of patients undergoing general and neuraxial anesthesia from electronic health records (EHRs) from 2016 to 2020. We first compared the deep neural network (DNN) with other models using the same input features to demonstrate effectiveness. Then, we combined the DNN model with bidirectional encoder representations from transformers (BERT) to extract information from clinical texts. The effects of adding text information on the model performance were compared using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Statistical significance was evaluated using P<.05. Results The final cohort contained 121,313 patients who underwent surgeries. A total of 1562 (1.29%) patients died within 30 days of surgery. Our BERT-DNN model achieved the highest AUROC (0.964, 95% CI 0.961-0.967) and AUPRC (0.336, 95% CI 0.276-0.402). The AUROC of the BERT-DNN was significantly higher compared to logistic regression (AUROC=0.952, 95% CI 0.949-0.955) and the American Society of Anesthesiologist Physical Status (ASAPS AUROC=0.892, 95% CI 0.887-0.896) but not significantly higher compared to the DNN (AUROC=0.959, 95% CI 0.956-0.962) and the random forest (AUROC=0.961, 95% CI 0.958-0.964). The AUPRC of the BERT-DNN was significantly higher compared to the DNN (AUPRC=0.319, 95% CI 0.260-0.384), the random forest (AUPRC=0.296, 95% CI 0.239-0.360), logistic regression (AUPRC=0.276, 95% CI 0.220-0.339), and the ASAPS (AUPRC=0.149, 95% CI 0.107-0.203). Conclusions Our BERT-DNN model has an AUPRC significantly higher compared to previously proposed models using no text and an AUROC significantly higher compared to logistic regression and the ASAPS. This technique helps identify patients with higher risk from the surgical description text in EHRs.

Prediction of Complications and Prognostication in Perioperative Medicine: A Systematic Review and PROBAST Assessment of Machine Learning Tools

“…28,29,32,36,100 One study predicted 30-day mortality risk related to myocardial injury in noncardiac surgery patients, 37 while another developed a natural language processing model using deep learning to analyze medical records and obtain diagnoses directly from notes written by a physician. 27–35,38,100…”

Large Language Model Capabilities in Perioperative Risk Prediction and Prognostication

“…When strictly comparing metrics such as F1 score, LLMs still underperform dedicated classification models utilizing tabular features. [45][46][47][48][49][50][51][52][53][54][55][56][57][58] Traditional machine learning models are rarely utilized in the clinical setting because of difficulty in interpreting a model's predictions. In contrast, LLMs present natural language explanations understandable to human clinicians, and can develop a rationale for each outcome variable of interest.…”

“…The LLM in this study exhibited very good performance at ASA-PS classification prediction, ICU admission prediction, and hospital mortality prediction. When strictly comparing metrics such as F1 score, LLMs still underperform dedicated classification models utilizing tabular features . Traditional machine learning models are rarely utilized in the clinical setting because of difficulty in interpreting a model’s predictions.…”

Understanding Views Around the Creation of a Consented, Donated Databank of Clinical Free Text to Develop and Train Natural Language Processing Models for Research: Focus Group Interviews With Stakeholders

“…There was a general agreement among stakeholder groups that the databank should draw upon a range of data sources to ensure that NLP tools reflect an integrated care system in the future, although there were mixed views about the benefits of linking to other data sources. The types of data to be included in the databank (eg, structured data in the GP or hospital record that may improve the performance of existing models owing to the inclusion of text-based features [ 34 , 35 ] or linkage to other data outside the NHS EHR, eg, national registry, mortality, or administrative data) should reflect stakeholder views on acceptability and practicalities, including cost, especially at the start. These issues should be explored in more detail in the next phase of the study.…”