Machine Learning Refinement of the NSQIP Risk Calculator: Who Survives the “Hail Mary” Case?

Abstract

BACKGROUND: The American College of Surgeons (ACS) NSQIP risk calculator helps guide operative decision making. In patients with significant surgical risk, it may be unclear whether to proceed with “Hail Mary”–type interventions. To refine predictions, a local interpretable model-agnostic explanations machine (LIME) learning algorithm was explored to determine weighted patient-specific factors’ contribution to mortality. STUDY DESIGN: The ACS-NSQIP database was queried for all surgical patients with mortality probability greater than 50% between 2012 and 2019. Preoperative factors (n = 38) were evaluated using stepwise logistic regression; 26 significant factors were used in gradient boosted machine (GBM) modeling. Data were divided into training and testing sets, and model performance was substantiated with 10-fold cross validation. LIME provided individual subject mortality. The GBM-trained model was interpolated to LIME, and predictions were made using the test dataset. RESULTS: There were 6,483 deaths (53%) among 12,248 admissions. GBM modeling displayed good performance (area under the curve = 0.65, 95% CI 0.636–0.671). The top 5 factors (% contribution) to mortality included: septic shock (27%), elevated International Normalized Ratio (22%), ventilator-dependence (14%), thrombocytopenia (14%), and elevated serum creatinine (5%). LIME modeling subset personalized patients by factors and weights on survival. In the entire cohort, mortality positive predictive value with 2 factor combinations was 53.5% (specificity 0.713), 3 combinations 64.2% (specificity 0.835), 4 combinations 72.1% (specificity 0.943), and all 5 combinations 77.9% (specificity 0.993). Conversely, mortality positive predictive value fell to 34% in the absence of 4 factors. CONCLUSIONS: Through the application of machine learning algorithms (GBM and LIME), our model individualized predicted mortality and contributing factors with substantial ACS-NSQIP predicted mortality. USE of machine learning techniques may better inform operative decisions and family conversations in cases of significant surgical risk.

Artificial Intelligence–enabled Decision Support in Surgery

“…Twentythree articles 9,19,21,[23][24][25]27,29,30,[32][33][34]36,37,39,41,42,[44][45][46][48][49][50] (63.9%) reported precision metrics (area under the precision-recall curve, positive predictive value, or F1 score). Twenty-five articles 9,20,21,[23][24][25][26][27][28]31,33,34,36,38,40,[42][43][44][45][46][47][48][49][50] (69.4%) included explainability mechanisms to convey the relative importance of input features in determining outputs. Thirteen articles 9,16,17,…”

“…Twenty-three articles9,19,21,23–25,27,29,30,32–34,36,37,39,41,42,44–46,48–50 (63.9%) reported precision metrics (area under the precision-recall curve, positive predictive value, or F1 score). Twenty-five articles9,20,21,23–28,31,33,34,36,38,40,42–50 (69.4%) included explainability mechanisms to convey the relative importance of input features in determining outputs. Thirteen articles9,16,17,20,25,27,29,30,35,38,45,46,50 (36.1%) presented a framework that could be used for clinical implementation; none of the articles assessed the efficacy of clinical implementation.…”

A machine learning approach to high‐risk cardiac surgery risk scoring

“…19 The application of machine learning modeling to NSQIP has similarly shown improvements in predicting outcomes. [20][21][22] The result of this increased interest in risk scoring optimization using advanced statistical modeling may serve to improve patient outcomes, though clinical adoption of these techniques remains in its infancy. 23 There are several limitations of this analysis that should be The application of a machine learning approach to high-risk cardiac surgery using GBM and LIME offers individualized predicted mortality and identification of significant features and influence on mortality.…”

“…NSQIP employs a hierarchical multivariable logistic regression modeling strategy for hospital performance risk adjustment to account for the grouping of patients within hospitals, reduce false‐positive rates through multiple sampling, and adjust for hospitals with limited case numbers by Bayesian shrinkage 19 . The application of machine learning modeling to NSQIP has similarly shown improvements in predicting outcomes 20–22 . The result of this increased interest in risk scoring optimization using advanced statistical modeling may serve to improve patient outcomes, though clinical adoption of these techniques remains in its infancy 23…”

Predicting Postoperative Mortality With Deep Neural Networks and Natural Language Processing: Model Development and Validation

“…Machine learning (ML) is widely applied to medical problems, including for predicting postoperative mortality [ 6 - 11 ]. ML models can automatically predict postoperative mortality using electronic health records (EHRs) before surgery, and they achieve a superior area under the receiver operating characteristic curve (AUROC) than previous methods [ 6 ].…”