The Application and Comparison of Machine Learning Models for the Prediction of Breast Cancer Prognosis: Retrospective Cohort Study

Abstract

Background Over the recent years, machine learning methods have been increasingly explored in cancer prognosis because of the appearance of improved machine learning algorithms. These algorithms can use censored data for modeling, such as support vector machines for survival analysis and random survival forest (RSF). However, it is still debated whether traditional (Cox proportional hazard regression) or machine learning-based prognostic models have better predictive performance. Objective This study aimed to compare the performance of breast cancer prognostic prediction models based on machine learning and Cox regression. Methods This retrospective cohort study included all patients diagnosed with breast cancer and subsequently hospitalized in Fudan University Shanghai Cancer Center between January 1, 2008, and December 31, 2016. After all exclusions, a total of 22,176 cases with 21 features were eligible for model development. The data set was randomly split into a training set (15,523 cases, 70%) and a test set (6653 cases, 30%) for developing 4 models and predicting the overall survival of patients diagnosed with breast cancer. The discriminative ability of models was evaluated by the concordance index (C-index), the time-dependent area under the curve, and D-index; the calibration ability of models was evaluated by the Brier score. Results The RSF model revealed the best discriminative performance among the 4 models with 3-year, 5-year, and 10-year time-dependent area under the curve of 0.857, 0.838, and 0.781, a D-index of 7.643 (95% CI 6.542, 8.930) and a C-index of 0.827 (95% CI 0.809, 0.845). The statistical difference of the C-index was tested, and the RSF model significantly outperformed the Cox-EN (elastic net) model (C-index 0.816, 95% CI 0.796, 0.836; P=.01), the Cox model (C-index 0.814, 95% CI 0.794, 0.835; P=.003), and the support vector machine model (C-index 0.812, 95% CI 0.793, 0.832; P<.001). The 4 models’ 3-year, 5-year, and 10-year Brier scores were very close, ranging from 0.027 to 0.094 and less than 0.1, which meant all models had good calibration. In the context of feature importance, elastic net and RSF both indicated that TNM staging, neoadjuvant therapy, number of lymph node metastases, age, and tumor diameter were the top 5 important features for predicting the prognosis of breast cancer. A final online tool was developed to predict the overall survival of patients with breast cancer. Conclusions The RSF model slightly outperformed the other models on discriminative ability, revealing the potential of the RSF method as an effective approach to building prognostic prediction models in the context of survival analysis.

Machine learning approaches for predicting 5‐year breast cancer survival: A multicenter study

“…In contrast, machine learning and deep learning studies focus on clinical data and their applications to patient‐level prediction for breast cancer are limited. Studies by Ganggayah et al, 17 Xiao et al, 18 and Huang et al 19 using machine learning algorithms to predict the overall survival of breast cancer patients showed comparable performance to our research. Although RF was not the best among those algorithms, it performed well in all four studies.…”

“…[37][38][39] As our study focused on the overall deaths of breast cancer patients, we took into consideration not only breast cancerspecific factors but also general health-related factors. Another important feature of our model was CCI score, a tool used for over [40][41][42][43] this variable was not considered in previous machine learning studies that had a similar aim to ours, [17][18][19]35 as these studies mainly focused on tumor characteristics. Hypertension, a comorbidity not included in the CCI, was another variable that contributed to the models' performance.…”

“…Online showed less accurate results in the high‐risk group of Taiwanese patients 16 . Most machine learning models focus on cancer characteristics such as lymph nodes, tumor size, and biomarkers 17–19 . To date, few models have considered the effects of comorbidities and long‐term medications on breast cancer prognosis.…”

Prediction of contraceptive discontinuation among reproductive-age women in Ethiopia using Ethiopian Demographic and Health Survey 2016 Dataset: A Machine Learning Approach

“…Feature importance is the average measure of how significant a feature is in comparison to other features used in the ensemble model to predict the outcome variable; higher feature importance means that that feature is used to differentiate one outcome versus the other more frequently [46]. This technique has been widely used in previous public health studies to identify predictors/factors of various health outcomes [46][47][48][49]. Furthermore, it has been demonstrated to be the best feature selection method when compared to other feature selection methods such as Boruta and recursive feature elimination techniques, which makes it extremely useful and efficient in selecting the important variables [50].…”

Modeling time to stop trachoma MDA in persistent districts of Ethiopia (Comparison of cox proportional hazard regression and machine learning models)

“…Harrell's concordance index (C-index), which calculates the proportion of observation pairs for which the model predictions and observed survival times agree [19]. Several authors have used C_index to make comparison among different survival models [19][20][21].Values range from 0.5 to 1, with a C index of 0 indicating a model with no discrimination and a C-Index of 1, a perfect model. As shown in Table 5, the entire model showed similar performance as measured by performance metrics, with nearly identical C indices.…”