Development and Validation of Machine Learning Algorithms to Predict 1-Year Ischemic Stroke and Bleeding Events in Patients with Atrial Fibrillation and Cancer

Abstract

In this study, we leveraged machine learning (ML) approach to develop and validate new assessment tools for predicting stroke and bleeding among patients with atrial fibrillation (AFib) and cancer. We conducted a retrospective cohort study including patients who were newly diagnosed with AFib with a record of cancer from the 2012–2018 Surveillance, Epidemiology, and End Results (SEER)-Medicare database. The ML algorithms were developed and validated separately for each outcome by fitting elastic net, random forest (RF), extreme gradient boosting (XGBoost), support vector machine (SVM), and neural network models with tenfold cross-validation (train:test = 7:3). We obtained area under the curve (AUC), sensitivity, specificity, and F2 score as performance metrics. Model calibration was assessed using Brier score. In sensitivity analysis, we resampled data using Synthetic Minority Oversampling Technique (SMOTE). Among 18,388 patients with AFib and cancer, 523 (2.84%) had ischemic stroke and 221 (1.20%) had major bleeding within one year after AFib diagnosis. In prediction of ischemic stroke, RF significantly outperformed other ML models [AUC (0.916, 95% CI 0.887–0.945), sensitivity 0.868, specificity 0.801, F2 score 0.375, Brier score = 0.035]. However, the performance of ML algorithms in prediction of major bleeding was low with highest AUC achieved by RF (0.623, 95% CI 0.554–0.692). RF models performed better than CHA2DS2-VASc and HAS-BLED scores. SMOTE did not improve the performance of the ML algorithms. Our study demonstrated a promising application of ML in stroke prediction among patients with AFib and cancer. This tool may be leveraged in assisting clinicians to identify patients at high risk of stroke and optimize treatment decisions.

Machine Learning - Based Bleeding Risk Predictions in Atrial Fibrillation Patients on Direct Oral Anticoagulants

“…Furthermore, the HAS-BLED score’s reliance on labile INR is outdated, given the shift towards DOACs 13 . Our findings of a modest improvement in ML performance over conventional risk scores align with prior studies 16,17 , which consistently demonstrated the poor performance of the HAS-BLED score, with AUC-ROC ranging from 0.50 to 0.64 16,19,20,44,45 . However, these studies were limited by their focus on broader contexts or specific AF subpopulations 18,19 , restricting their applicability to the broader AF population on DOACs in a real-world clinical scenario when first evaluated by a cardiologist for AF management.…”

“…Our findings of a modest improvement in ML performance over conventional risk scores align with prior studies 16,17 , which consistently demonstrated the poor performance of the HAS-BLED score, with AUC-ROC ranging from 0.50 to 0.64 16,19,20,44,45 . However, these studies were limited by their focus on broader contexts or specific AF subpopulations 18,19 , restricting their applicability to the broader AF population on DOACs in a real-world clinical scenario when first evaluated by a cardiologist for AF management. It is worth noting that in the original HAS-BLED publication, the AUC for the derivation cohort was 0.72, and for the validation cohort, it ranged from 0.50 to 0.67 among patients on warfarin.…”

“…Previous studies have explored the use of machine learning (ML) models in predicting bleeding risk in AF patients, demonstrating improved performance over conventional scoring systems 16,17 . However, these studies have limitations, such as relying on registry-based data that may not accurately represent the complexity and heterogeneity of real-world clinical practice, focusing on broader contexts like predicting bleeding risk from antithrombotic therapy in general patient populations, or concentrating only on AF subpopulations 18,19 . Moreover, they lack direct comparison between ML models and multiple conventional risk scores, with most studies comparing ML models to only individual risk scores, such as HAS-BLED 20 .…”