Performance and usability of pre‐operative prediction models for 30‐day peri‐operative mortality risk: a systematic review

Abstract

Estimating pre-operative mortality risk may inform clinical decision-making for peri-operative care. However, pre-operative mortality risk prediction models are rarely implemented in routine clinical practice. High predictive accuracy and clinical usability are essential for acceptance and clinical implementation. In this systematic review, we identified and appraised prediction models for 30-day postoperative mortality in noncardiac surgical cohorts. PubMed and Embase were searched up to December 2022 for studies investigating pre-operative prediction models for 30-day mortality. We assessed predictive performance in terms of discrimination and calibration. Risk of bias was evaluated using a tool to assess the risk of bias and applicability of prediction model studies. To further inform potential adoption, we also assessed clinical usability for selected models. In all, 15 studies evaluating 10 prediction models were included. Discrimination ranged from a cstatistic of 0.82 (MySurgeryRisk) to 0.96 (extreme gradient boosting machine learning model). Calibration was reported in only six studies. Model performance was highest for the surgical outcome risk tool (SORT) and its external validations. Clinical usability was highest for the surgical risk pre-operative assessment system. The SORT and risk quantification index also scored high on clinical usability. We found unclear or high risk of bias in the development of all models. The SORT showed the best combination of predictive performance and clinical usability and has been externally validated in several heterogeneous cohorts. To improve clinical uptake, full integration of reliable models with sufficient face validity within the electronic health record is imperative.

When (not) to apply clinical risk prediction models to improve patient care

“…Safe and effective clinical risk communication and management also require the clinician to feel confident in their risk prediction methodology, while maintaining a respectful understanding of the limits of that methodology. The systematic review by Vernooij et al [3] in this issue of Anaesthesia evaluates risk-prediction models to help clinicians choose the best of the available options, thus supporting the first of these challengeschoosing the right tool.…”

“…While discrimination is an important measure of validity, risk estimates also need to be reliable or calibrated [4]. In the review by Vernooij et al, discrimination was the ability of the clinical prediction model to differentiate patients who died or survived within 30 days after surgery using a concordance index that ranged from 0.50 (no discrimination) to 1.0 (perfect discrimination) [3].…”

“…For the most studied prognostic model for 30-day mortality [3], we estimated a summary concordance index for the surgical outcome risk tool (SORT) models to be high (0.91, 95% CI 0.90-0.92).…”

“…A hierarchy of analytic methods exists for assessing calibration ranging from weak (intercept and calibration slope), moderate (calibration curve showing estimated risk of death on the x-axis and observed deaths on the y-axis) to strong (plotting calibration in patients with similar patterns of covariates) [4]. Of the 15 studies, only six used various weak to moderate approaches for assessing the calibration [3] or reported the Hosmer-Lemeshow test p value that has known limitations as a calibration test [4]. Nevertheless, the observed to expected (O:E) risk ratio is commonly reported or can be extracted from the total number of the observed and expected outcome data from a Hosmer-Lemeshow goodness-of-fit table published in a primary study.…”

“…Good calibration can be defined as an O:E ratio between 0.80 and 1.20 [13]. We performed a meta-analysis on the calibration performance of prognostic models for postoperative 30-day mortality using JASP software version 0.16.4 (JASP Team, University of Amsterdam, The Netherlands) using studies included in the review by Vernooij et al [3]. The summary O:E ratio estimate and prediction interval suggest that SORT [15][16][17], surgical mortality probability model [18] and risk quantification index [19] prognostic models were likely to be overall well calibrated (Fig.…”