Development and Validation of an Explainable Machine Learning Model for Major Complications After Cytoreductive Surgery

Deng, Hongbo; Eftekhari, Zahra; Carlin, Cameron; Veerapong, Jula; Fournier, Keith F.; Johnston, Fabian M.; Dineen, Seán; Powers, Benjamin D.; Hendrix, Ryan J.; Lambert, Laura; Abbott, Daniel E.; Walle, Kara Vande; Grotz, Travis E.; Patel, Sameer H.; Clarke, Callisia N.; Staley, Charles A.; Abdel‐Misih, Sherif; Cloyd, Jordan M.; Lee, Byrne; Fong, Yuman; Raoof, Mustafa

doi:10.1001/jamanetworkopen.2022.12930

Cited by 18 publications

(14 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…None of the articles described equity analyses in which model performance was stratified by sex or race. Twenty-three articles9,19,21,23–25,27,29,30,32–34,36,37,39,41,42,44–46,48–50 (63.9%) reported precision metrics (area under the precision-recall curve, positive predictive value, or F1 score). Twenty-five articles9,20,21,23–28,31,33,34,36,38,40,42–50 (69.4%) included explainability mechanisms to convey the relative importance of input features in determining outputs.…”

Section: Resultsmentioning

confidence: 99%

“…Twenty-three articles9,19,21,23–25,27,29,30,32–34,36,37,39,41,42,44–46,48–50 (63.9%) reported precision metrics (area under the precision-recall curve, positive predictive value, or F1 score). Twenty-five articles9,20,21,23–28,31,33,34,36,38,40,42–50 (69.4%) included explainability mechanisms to convey the relative importance of input features in determining outputs. Thirteen articles9,16,17,20,25,27,29,30,35,38,45,46,50 (36.1%) presented a framework that could be used for clinical implementation; none of the articles assessed the efficacy of clinical implementation.…”

Section: Resultsmentioning

confidence: 99%

“…Twentythree articles 9,19,21,[23][24][25]27,29,30,[32][33][34]36,37,39,41,42,[44][45][46][48][49][50] (63.9%) reported precision metrics (area under the precision-recall curve, positive predictive value, or F1 score). Twenty-five articles 9,20,21,[23][24][25][26][27][28]31,33,34,36,38,40,[42][43][44][45][46][47][48][49][50] (69.4%) included explainability mechanisms to convey the relative importance of input features in determining outputs. Thirteen articles 9,16,17,…”

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

Artificial Intelligence–enabled Decision Support in Surgery

et al. 2023

View full text Add to dashboard Cite

Objective: To summarize state-of-the-art artificial intelligence-enabled decision support in surgery and to quantify deficiencies in scientific rigor and reporting. Background: To positively affect surgical care, decision-support models must exceed current reporting guideline requirements by performing external and real-time validation, enrolling adequate sample sizes, reporting model precision, assessing performance across vulnerable populations, and achieving clinical implementation; the degree to which published models meet these criteria is unknown. Methods: Embase, PubMed, and MEDLINE databases were searched from their inception to September 21, 2022 for articles describing artificial intelligence-enabled decision support in surgery that uses preoperative or intraoperative data elements to predict complications within 90 days of surgery. Scientific rigor and reporting criteria were assessed and reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. Results: Sample size ranged from 163-2,882,526, with 8/36 articles (22.2%) featuring sample sizes of less than 2000; 7 of these 8 articles (87.5%) had below-average (< 0.83) area under the receiver operating characteristic or accuracy. Overall, 29 articles (80.6%) performed internal validation only, 5 (13.8%) performed external validation, and 2 (5.6%) performed real-time validation. Twenty-three articles (63.9%) reported precision. No articles reported performance across sociodemographic categories. Thirteen articles (36.1%) presented a framework that could be used for clinical implementation; none assessed clinical implementation efficacy. Conclusions: Artificial intelligence-enabled decision support in surgery is limited by reliance on internal validation, small sample sizes that risk overfitting and sacrifice predictive performance, and failure to report confidence intervals, precision, equity analyses, and clinical implementation. Researchers should strive to improve scientific quality.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Artificial Intelligence–enabled Decision Support in Surgery

et al. 2023

View full text Add to dashboard Cite

show abstract

“…The shapley additive explanations (SHAP) is an artificial intelligence strategy based on game theory, which provides a unified method to interpreting machine learning models (20)(21)(22). It can be used to unlock the intrinsic importance of features for the prediction, such as, treatment decision-making.…”

Section: Introductionmentioning

confidence: 99%

“…Thus, risk assessment from objective evidence is particularly important for clinical decision-making. The artificial intelligence annotated model is widely accepted to provide assistance for clinical decision-making ( 20 , 21 ). And there remains an opportunity to augment decision-making through artificial intelligence annotated tools for patients with advanced GC.…”

Section: Introductionmentioning

confidence: 99%

Artificial intelligence annotated clinical-pathologic risk model to predict outcomes of advanced gastric cancer

Chen¹,

Lin²,

Ying³

et al. 2023

Front. Oncol.

View full text Add to dashboard Cite

BackgroundGastric cancer with synchronous distant metastases indicates a dismal prognosis. The success in survival improvement mainly relies on our ability to predict the potential benefit of a therapy. Our objective is to develop an artificial intelligence annotated clinical-pathologic risk model to predict its outcomes.MethodsIn participants (n=47553) with gastric cancer of the surveillance, epidemiology, and end results program, we selected patients with distant metastases at first diagnosis, complete clinical-pathologic data and follow-up information. Patients were randomly divided into the training and test cohort at 7:3 ratio. 93 patients with advanced gastric cancer from six other cancer centers were collected as the external validation cohort. Multivariable analysis was used to identify the prognosis-related clinical-pathologic features. Then a survival prediction model was established and validated. Importantly, we provided explanations to the prediction with artificial intelligence SHAP (Shapley additive explanations) method. We also provide novel insights into treatment options.ResultsData from a total 2549 patients were included in model development and internal test (median age, 61 years [range, 53-69 years]; 1725 [67.7%] male). Data from an additional 93 patients were collected as the external validation cohort (median age, 59 years [range, 48-66 years]; 51 [54.8%] male). The clinical-pathologic model achieved a consistently high accuracy for predicting prognosis in the training (C-index: 0.705 [range, 0.690-0.720]), test (C-index: 0.737 [range, 0.717-0.757]), and external validation (C-index: 0.694 [range, 0.562-0.826]) cohorts. Shapley values indicated that undergoing surgery, chemotherapy, young, absence of lung metastases and well differentiated were the top 5 contributors to the high likelihood of survival. A combination of surgery and chemotherapy had the greatest benefit. However, aggressive treatment did not equate to a survival benefit. SHAP dependence plots demonstrated insightful nonlinear interactive associations among predictors in survival benefit prediction. For example, patients who were elderly, or poor differentiated, or presence of lung or bone metastases had a worse prognosis if they undergo surgery or chemotherapy, while patients with metastases to liver alone seemed to gain benefit from surgery and chemotherapy.ConclusionIn this large multicenter cohort study, we developed an artificial intelligence annotated clinical-pathologic risk model to predict outcomes of advanced gastric cancer. It could be used to discuss treatment options.

show abstract

Explainable machine learning model for pre‐frailty risk assessment in community‐dwelling older adults

Du,

Zhang,

Liu

et al. 2024

Health Care Science

View full text Add to dashboard Cite

BackgroundFrailty in older adults is linked to increased risks and lower quality of life. Pre‐frailty, a condition preceding frailty, is intervenable, but its determinants and assessment are challenging. This study aims to develop and validate an explainable machine learning model for pre‐frailty risk assessment among community‐dwelling older adults.MethodsThe study included 3141 adults aged 60 or above from the China Health and Retirement Longitudinal Study. Pre‐frailty was characterized by one or two criteria from the physical frailty phenotype scale. We extracted 80 distinct features across seven dimensions to evaluate pre‐frailty risk. A model was constructed using recursive feature elimination and a stacking‐CatBoost distillation module on 80% of the sample and validated on a separate 20% holdout data set.ResultsThe study used data from 2508 community‐dwelling older adults (mean age, 67.24 years [range, 60–96]; 1215 [48.44%] females) to develop a pre‐frailty risk assessment model. We selected 57 predictive features and built a distilled CatBoost model, which achieved the highest discrimination (AUROC: 0.7560 [95% CI: 0.7169, 0.7928]) on the 20% holdout data set. The living city, BMI, and peak expiratory flow (PEF) were the three most significant contributors to pre‐frailty risk. Physical and environmental factors were the top 2 impactful feature dimensions.ConclusionsAn accurate and interpretable pre‐frailty risk assessment framework using state‐of‐the‐art machine learning techniques and explanation methods has been developed. Our framework incorporates a wide range of features and determinants, allowing for a comprehensive and nuanced understanding of pre‐frailty risk.

show abstract

Development and Validation of an Explainable Machine Learning Model for Major Complications After Cytoreductive Surgery

Cited by 18 publications

References 23 publications

Artificial Intelligence–enabled Decision Support in Surgery

Artificial Intelligence–enabled Decision Support in Surgery

Artificial intelligence annotated clinical-pathologic risk model to predict outcomes of advanced gastric cancer

Explainable machine learning model for pre‐frailty risk assessment in community‐dwelling older adults

Contact Info

Product

Resources

About