Commentary: Artificial Intelligence and Statistics: Just the Old Wine in New Wineskins?

Hond, Anne de; Calster, Ben Van

doi:10.3389/fdgth.2022.923944

Cited by 2 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We measure the predictive performance for all validation moments along three axes: discrimination, calibration, and net benefit. Discrimination quantifies the separation between low- and high-risk subjects and was measured via the AUC ( 23 ). The AUC ranges between 0.5 and 1, with higher values indicating better discrimination.…”

Section: Methodsmentioning

confidence: 99%

“…The AUC ranges between 0.5 and 1, with higher values indicating better discrimination. Calibration is good when the proportion of patients receiving a given risk score approximates that risk score (e.g., 40% of patients are readmitted within the group of patients receiving a 40% risk of readmission) (23). Calibration was assessed through the calibration slope (1 for perfect calibration), intercept (0 for perfect calibration), and calibration loss by bins (lower loss is better) (21, 24, 25).…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model

Hond

Kant

Fornasa³

et al. 2022

Critical Care Medicine

Self Cite

View full text Add to dashboard Cite

OBJECTIVES: Many machine learning (ML) models have been developed for application in the ICU, but few models have been subjected to external validation. The performance of these models in new settings therefore remains unknown. The objective of this study was to assess the performance of an existing decision support tool based on a ML model predicting readmission or death within 7 days after ICU discharge before, during, and after retraining and recalibration. DESIGN:A gradient boosted ML model was developed and validated on electronic health record data from 2004 to 2021. We performed an independent validation of this model on electronic health record data from 2011 to 2019 from a different tertiary care center. SETTING: Two ICUs in tertiary care centers in The Netherlands. PATIENTS:Adult patients who were admitted to the ICU and stayed for longer than 12 hours. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS:We assessed discrimination by area under the receiver operating characteristic curve (AUC) and calibration (slope and intercept). We retrained and recalibrated the original model and assessed performance via a temporal validation design. The final retrained model was cross-validated on all data from the new site. Readmission or death within 7 days after ICU discharge occurred in 577 of 10,052 ICU admissions (5.7%) at the new site. External validation revealed moderate discrimination with an AUC of 0.72 (95% CI 0.67-0.76). Retrained models showed improved discrimination with AUC 0.79 (95% CI 0.75-0.82) for the final validation model. Calibration was poor initially and good after recalibration via isotonic regression. CONCLUSIONS:In this era of expanding availability of ML models, external validation and retraining are key steps to consider before applying ML models to new settings. Clinicians and decision-makers should take this into account when considering applying new ML models to their local settings.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model

Hond

Kant

Fornasa³

et al. 2022

Critical Care Medicine

Self Cite

View full text Add to dashboard Cite

show abstract

“…Accuracy measures the proportion of correct predictions out of all predictions [ 59 , 60 ]. Precision quantifies the proportion of true positive predictions out of all positive predictions [ 61 , 62 ]. Recall (also known as sensitivity) indicates the proportion of true positive predictions out of all actual positive instances [ 63 ].…”

Section: Introductionmentioning

confidence: 99%

An Ensemble Classifiers for Improved Prediction of Native–Non-Native Protein–Protein Interaction

Pratiwi,

Tayara,

Chong

2024

IJMS

View full text Add to dashboard Cite

In this study, we present an innovative approach to improve the prediction of protein–protein interactions (PPIs) through the utilization of an ensemble classifier, specifically focusing on distinguishing between native and non-native interactions. Leveraging the strengths of various base models, including random forest, gradient boosting, extreme gradient boosting, and light gradient boosting, our ensemble classifier integrates these diverse predictions using a logistic regression meta-classifier. Our model was evaluated using a comprehensive dataset generated from molecular dynamics simulations. While the gains in AUC and other metrics might seem modest, they contribute to a model that is more robust, consistent, and adaptable. To assess the effectiveness of various approaches, we compared the performance of logistic regression to four baseline models. Our results indicate that logistic regression consistently underperforms across all evaluated metrics. This suggests that it may not be well-suited to capture the complex relationships within this dataset. Tree-based models, on the other hand, appear to be more effective for problems involving molecular dynamics simulations. Extreme gradient boosting (XGBoost) and light gradient boosting (LightGBM) are optimized for performance and speed, handling datasets effectively and incorporating regularizations to avoid over-fitting. Our findings indicate that the ensemble method enhances the predictive capability of PPIs, offering a promising tool for computational biology and drug discovery by accurately identifying potential interaction sites and facilitating the understanding of complex protein functions within biological systems.

show abstract

Commentary: Artificial Intelligence and Statistics: Just the Old Wine in New Wineskins?

Cited by 2 publications

References 10 publications

Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model

Predicting Readmission or Death After Discharge From the ICU: External Validation and Retraining of a Machine Learning Model

An Ensemble Classifiers for Improved Prediction of Native–Non-Native Protein–Protein Interaction

Contact Info

Product

Resources

About