Artificial intelligence versus surgeon gestalt in predicting risk of emergency general surgery

Abstract

BACKGROUND:Artificial intelligence (AI) risk prediction algorithms such as the smartphone-available Predictive OpTimal Trees in Emergency Surgery Risk (POTTER) for emergency general surgery (EGS) are superior to traditional risk calculators because they account for complex nonlinear interactions between variables, but how they compare to surgeons' gestalt remains unknown. Herein, we sought to: (1) compare POTTER to surgeons' surgical risk estimation and (2) assess how POTTER influences surgeons' risk estimation. STUDY DESIGN:A total of 150 patients who underwent EGS at a large quaternary care center between May 2018 and May 2019 were prospectively followed up for 30-day postoperative outcomes (mortality, septic shock, ventilator dependence, bleeding requiring transfusion, pneumonia), and clinical cases were systematically created representing their initial presentation. POTTER's outcome predictions for each case were also recorded. Thirty acute care surgeons with diverse practice settings and levels of experience were then randomized into two groups: 15 surgeons (SURG) were asked to predict the outcomes without access to POTTER's predictions while the remaining 15 (SURG-POTTER) were asked to predict the same outcomes after interacting with POTTER. Comparing to actual patient outcomes, the area under the curve (AUC) methodology was used to assess the predictive performance of (1) POTTER versus SURG, and (2) SURG versus SURG-POTTER. RESULTS:POTTER outperformed SURG in predicting all outcomes (mortality-AUC: 0.880 vs. 0.841; ventilator dependence-AUC: 0.928 vs. 0.833; bleeding-AUC: 0.832 vs. 0.735; pneumonia-AUC: 0.837 vs. 0.753) except septic shock (AUC: 0.816 vs. 0.820). SURG-POTTER outperformed SURG in predicting mortality (AUC: 0.870 vs. 0.841), bleeding (AUC: 0.811 vs. 0.735), pneumonia (AUC: 0.803 vs. 0.753) but not septic shock (AUC: 0.712 vs. 0.820) or ventilator dependence (AUC: 0.834 vs. 0.833). CONCLUSION:The AI risk calculator POTTER outperformed surgeons' gestalt in predicting the postoperative mortality and outcomes of EGS patients, and when used, improved the individual surgeons' risk prediction. Artificial intelligence algorithms, such as POTTER, could prove useful as a bedside adjunct to surgeons when preoperatively counseling patients.

Artificial intelligence in surgery

“…One notable example is the smartphone app-based POTTER calculator, which uses optimal classification trees and outperforms most existing mortality predictors with an accuracy of 0.92 at internal validation 52 , 0.93 in an external emergency surgery context 48 and 0.80 in an external validation cohort of patients >65 years of age receiving emergency surgery 47 . Notably, POTTER also showed improved predictive accuracy compared with surgeon gestalt 53 . Deep learning methods have also shown utility in neonatal cardiac transplantation outcomes, with high accuracy for predicting mortality and length of stay (AOROC values of 0.95 and 0.94, respectively) 54 .…”

What is the role of artificial intelligence in general surgery?

“…Risk prediction for mortality, bleeding, and pneumonia improved when surgeons used POTTER, although there was no significant improvement for septic shock or ventilator dependence. The AUC was calculated to evaluate the predictive performance of surgeons who used POTTER compared to those who did not [29].…”

Predicting Risk of Post-Operative Morbidity and Mortality following Gynaecological Oncology Surgery (PROMEGO): A Global Gynaecological Oncology Surgical Outcomes Collaborative Led Study

“…There is considerable uncertainty in health care, and risk prediction plays a fundamental role in a surgeon’s ability to drive clinical decisions, counsel patients, and evaluate outcomes. Studies have shown that clinicians are imperfect when predicting medical and surgical risk and often rely on their experience and subjective global assessment of patient fitness for surgery [ 1 , 2 , 3 , 4 ]. Surgical risk calculators are a set of tools with the potential to mitigate the highly variable perception of patient risk [ 5 , 6 , 7 , 8 ].…”