Highlighting uncertainty in clinical risk prediction using a model of emergency laparotomy mortality risk

Abstract

Clinical prediction models typically make point estimates of risk. However, values of key variables are often missing during model development or at prediction time, meaning that the point estimates mask significant uncertainty and can lead to over-confident decision making. We present a model of mortality risk in emergency laparotomy which instead presents a distribution of predicted risks, highlighting the uncertainty over the risk of death with an intuitive visualisation. We developed and validated our model using data from 127134 emergency laparotomies from patients in England and Wales during 2013–2019. We captured the uncertainty arising from missing data using multiple imputation, allowing prospective, patient-specific imputation for variables that were frequently missing. Prospective imputation allows early prognostication in patients where these variables are not yet measured, accounting for the additional uncertainty this induces. Our model showed good discrimination and calibration (95% confidence intervals: Brier score 0.071–0.078, C statistic 0.859–0.873, calibration error 0.031–0.059) on unseen data from 37 hospitals, consistently improving upon the current gold-standard model. The dispersion of the predicted risks varied significantly between patients and increased where prospective imputation occurred. We present a case study that illustrates the potential impact of uncertainty quantification on clinical decision making. Our model improves mortality risk prediction in emergency laparotomy and has the potential to inform decision-makers and assist discussions with patients and their families. Our analysis code was robustly developed and is publicly available for easy replication of our study and adaptation to predicting other outcomes.

When (not) to apply clinical risk prediction models to improve patient care

“…Of the prognostic models for 30-day mortality in the review by Vernooji et al [3], SORT has the greatest accuracy in heterogeneous surgical populations. However, there remains a need for clinicians to be cognisant of error margins, particularly at higher ends of risk [10]. Using models such as the SORT to inform clinical and shared decision-making about pathway decisions is recommended.…”

“…It is the reason we get out of bed in the morning – to do our best for the patient, considering their known risks for surgery and surgical outcome. Furthermore, the wide dispersion of the expected risk estimate from a clinical prediction model from missing or unmeasured variables at the time of prediction is commensurate with uncertainty [10]. This is particularly true at the higher ends of risk where, arguably, clinical decision‐making has a greater impact on patient pathways and outcomes.…”

“…This is particularly true at the higher ends of risk where, arguably, clinical decision‐making has a greater impact on patient pathways and outcomes. Put simply, an individual with a predicted 10% risk of mortality may have a higher or lower ‘true’ risk of mortality based on factors not considered in the prognostic model [10], and because the models themselves have been generated using data from fewer high‐risk than low‐risk patients.…”

“…Second, low compliance with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines [9] hinders the translation of evidence. Third, there are limited publicly available bedside risk calculators developed using a low burden of predictor collection for easy replication [8,10]. Finally, ``collective statistical illiteracy´´may cause difficulties for clinicians to interpret health statistics and explain them in a way that patients can comprehend [11].…”

Assessing risk in emergency laparotomy

“…It is a fundamental characteristic of population-based models that even the best-calibrated risk model will only ever provide point estimates that mask significant uncertainty. To overcome the challenges associated with applying population risk to individuals, Mathzig-Lee et al applied a machine-learning approach to NELA data [14]. They describe a model that accounts for unmeasured variables and may better support shared decision-making by reporting a distribution of predicted risks around a point estimate.…”