An ensemble weighted average conservative multi-fidelity surrogate modeling method for engineering optimization

Hu, Jiexiang; Peng, Yutong; Lin, Qian; Liu, Huaping; Zhou, Qi

doi:10.1007/s00366-020-01203-8

Cited by 8 publications

(2 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study heeds the call for additional metrics to address the lack of sensitivity of the most commonly used C-statistic and calibration slope in capturing the advantage of ML models [ 58 ]; we demonstrated the use of a consensus score [ 22 35 59-61 undefined undefined ] named CEM to take into account numerous metrics that have been found to be beneficial, covering overall accuracy [ 58 ], discrimination, calibration, and clinical utility. We wanted to analyze model performance across multiple metrics across time in this study.…”

Section: Discussionmentioning

confidence: 99%

Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

Dong,

Sinha,

Zhai

et al. 2024

JMIRx Med

View full text Add to dashboard Cite

Background The Society of Thoracic Surgeons and European System for Cardiac Operative Risk Evaluation (EuroSCORE) II risk scores are the most commonly used risk prediction models for in-hospital mortality after adult cardiac surgery. However, they are prone to miscalibration over time and poor generalization across data sets; thus, their use remains controversial. Despite increased interest, a gap in understanding the effect of data set drift on the performance of machine learning (ML) over time remains a barrier to its wider use in clinical practice. Data set drift occurs when an ML system underperforms because of a mismatch between the data it was developed from and the data on which it is deployed. Objective In this study, we analyzed the extent of performance drift using models built on a large UK cardiac surgery database. The objectives were to (1) rank and assess the extent of performance drift in cardiac surgery risk ML models over time and (2) investigate any potential influence of data set drift and variable importance drift on performance drift. Methods We conducted a retrospective analysis of prospectively, routinely gathered data on adult patients undergoing cardiac surgery in the United Kingdom between 2012 and 2019. We temporally split the data 70:30 into a training and validation set and a holdout set. Five novel ML mortality prediction models were developed and assessed, along with EuroSCORE II, for relationships between and within variable importance drift, performance drift, and actual data set drift. Performance was assessed using a consensus metric. Results A total of 227,087 adults underwent cardiac surgery during the study period, with a mortality rate of 2.76% (n=6258). There was strong evidence of a decrease in overall performance across all models (P<.0001). Extreme gradient boosting (clinical effectiveness metric [CEM] 0.728, 95% CI 0.728-0.729) and random forest (CEM 0.727, 95% CI 0.727-0.728) were the overall best-performing models, both temporally and nontemporally. EuroSCORE II performed the worst across all comparisons. Sharp changes in variable importance and data set drift from October to December 2017, from June to July 2018, and from December 2018 to February 2019 mirrored the effects of performance decrease across models. Conclusions All models show a decrease in at least 3 of the 5 individual metrics. CEM and variable importance drift detection demonstrate the limitation of logistic regression methods used for cardiac surgery risk prediction and the effects of data set drift. Future work will be required to determine the interplay between ML models and whether ensemble models could improve on their respective performance advantages.

show abstract

Section: Discussionmentioning

confidence: 99%

Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

Dong,

Sinha,

Zhai

et al. 2024

JMIRx Med

View full text Add to dashboard Cite

show abstract

“…[2] While calibration drift over time is well documented amongst EuroSCORE and logistic regression models for hospital mortality, the susceptibility of competing ML modelling methods to dataset drift has not been well studied in cardiac surgery. [50] This study heeds to the call for additional metrics to address the lack of sensitivity of the most commonly used C-statistic and calibration slope in capturing the advantage of ML models,[51] by demonstrating the use of a consensus score [19,[52][53][54][55] named CEM to take into account numerous metrics that have been found to be beneficial, covering overall accuracy,[51] discrimination, calibration and clinical utility. This study showed invariance in model ranking for the CEM in both temporal and non-temporal analyses, indicating there is value for this consensus scoring approach in performance drift evaluation.…”

Section: Discussionmentioning

confidence: 99%

Performance drift is a major barrier to the safe use of machine learning in cardiac surgery

Dong

Sinha

Zhai

et al. 2023

Preprint

View full text Add to dashboard Cite

Objectives The Society of Thoracic Surgeons (STS), and EuroSCORE II (ES II) risk scores, are the most commonly used risk prediction models for adult cardiac surgery post-operative in-hospital mortality. However, they are prone to miscalibration over time, and poor generalisation across datasets and their use remain controversial. It has been suggested that using Machine Learning (ML) techniques, a branch of Artificial intelligence (AI), may improve the accuracy of risk prediction. Despite increased interest, a gap in understanding the effect of dataset drift on the performance of ML over time remains a barrier to its wider use in clinical practice. Dataset drift occurs when a machine learning system underperforms because of a mismatch between the dataset it was developed and the data on which it is deployed. Here we analyse this potential concern in a large United Kingdom (UK) database. Methods: A retrospective analyses of prospectively routinely gathered data on adult patients undergoing cardiac surgery in the UK between 2012-2019. We temporally split the data 70:30 into a training and validation subset. ES II and five ML mortality prediction models were assessed for relationships between and within variable importance drift, performance drift and actual dataset drift using temporal and non-temporal invariant consensus scoring, combining geometric average results of all metrics as the Clinical Effective Metric (CEM). Results: A total of 227,087 adults underwent cardiac surgery during the study period with a mortality rate of 2.76%. There was a strong evidence of decrease in overall performance across all models (p < 0.0001). Xgboost (CEM 0.728 95CI: 0.728-0.729) and Random Forest (CEM 0.727 95CI 0.727-0.728) were the best overall performing models both temporally and non-temporally. ES II perfomed worst across all comparisons. Sharp changes in variable importance and dataset drift between 2017-10 to 2017-12, 2018-06 to 2018-07 and 2018-12 to 2019-02 mirrored effects of performance decrease across models. Conclusions: Combining the metrics covering all four aspects of discrimination, calibration, clinical usefulness and overall accuracy into a single consensus metric improved the efficiency of cognitive decision-making. All models show a decrease in at least 3 of the 5 individual metrics. CEM and variable importance drift detection demonstrate the limitation of logistic regression methods used for cardiac surgery risk prediction and the effects of dataset drift. Future work will be required to determine the interplay between ML and whether ensemble models could take advantage of their respective performance advantages.

show abstract

SBSC+SRU: an error-guided adaptive Kriging method for expensive system reliability analysis

Cheng

2022

Struct Multidisc Optim

View full text Add to dashboard Cite

An ensemble weighted average conservative multi-fidelity surrogate modeling method for engineering optimization

Cited by 8 publications

References 41 publications

Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

Performance drift is a major barrier to the safe use of machine learning in cardiac surgery

SBSC+SRU: an error-guided adaptive Kriging method for expensive system reliability analysis

Contact Info

Product

Resources

About