Robust performance of deep learning for distinguishing glioblastoma from single brain metastasis using radiomic features: model development and validation

Bae, Soong June; An, Chansik; Ahn, Sung Soo; Kim, Hwiyoung; Han, Kyunghwa; Kim, Sang Wook; Park, Ji Eun; Kim, Ho Sung; Lee, Seung Koo

doi:10.1038/s41598-020-68980-6

Cited by 75 publications

(87 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neurosurgical examples include machine learning algorithms for glioblastoma, deep brain stimulation, traumatic brain injury, stroke, and spine surgery [38,[44][45][46][47][48][49][50][51][52][53]. Deep learning algorithms are also increasingly being used to further improve the WHO 2016 classification of high-grade gliomas via histological and biomolecular variables for more concise diagnosis and classification of gliomas [54][55][56].…”

Section: Discussionmentioning

confidence: 99%

Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential

et al. 2020

View full text Add to dashboard Cite

Glioblastoma is associated with a poor prognosis. Even though survival statistics are well-described at the population level, it remains challenging to predict the prognosis of an individual patient despite the increasing number of prognostic models. The aim of this study is to systematically review the literature on prognostic modeling in glioblastoma patients. A systematic literature search was performed to identify all relevant studies that developed a prognostic model for predicting overall survival in glioblastoma patients following the PRISMA guidelines. Participants, type of input, algorithm type, validation, and testing procedures were reviewed per prognostic model. Among 595 citations, 27 studies were included for qualitative review. The included studies developed and evaluated a total of 59 models, of which only seven were externally validated in a different patient cohort. The predictive performance among these studies varied widely according to the AUC (0.58–0.98), accuracy (0.69–0.98), and C-index (0.66–0.70). Three studies deployed their model as an online prediction tool, all of which were based on a statistical algorithm. The increasing performance of survival prediction models will aid personalized clinical decision-making in glioblastoma patients. The scientific realm is gravitating towards the use of machine learning models developed on high-dimensional data, often with promising results. However, none of these models has been implemented into clinical care. To facilitate the clinical implementation of high-performing survival prediction models, future efforts should focus on harmonizing data acquisition methods, improving model interpretability, and externally validating these models in multicentered, prospective fashion.

show abstract

Section: Discussionmentioning

confidence: 99%

Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Two classi cation tasks of different di culty levels were performed using radiomics features extracted from brain magnetic resonance imaging (MRI). The rst task was a "simple" task of differentiating between glioblastoma (GBM) and single metastasis; the accuracy of radiomics-based ML for this task has been reported to be up to 90% (12,13). In the current study, the rst task dataset consisted of 167 adult patients with pathologically con rmed single GBM (n=109) or brain metastasis (n=58) following brain MRI between January 2014 and December 2017.…”

Section: Subjectsmentioning

confidence: 99%

“…The second task dataset consisted of 258 adult patients with a low-(n=163) or high-grade (n=95) meningiomas diagnosed between February 2008 and September 2018. Both the datasets were taken from the same tertiary academic hospital; some subsets of these patients were used in previous studies (12,16). Additionally, two undersampled datasets were created by randomly sampling 50% of the datasets to determine the effect of sample size.…”

Section: Subjectsmentioning

confidence: 99%

Radiomics machine learning study with small sample size: single random training-test set split may result in unreliable results

Park

Ahn

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Objective: This study aims to determine how randomly splitting a dataset into training and test sets affects the estimated performance of a machine learning model under different conditions, using real-world brain tumor radiomics data.Materials and Methods: We conducted two classification tasks of different difficulty levels with magnetic resonance imaging (MRI) radiomics features: (1) “Simple” task, glioblastomas [n=109] vs. brain metastasis [n=58] and (2) “difficult” task, low- [n=163] vs. high-grade [n=95] meningiomas. Additionally, two undersampled datasets were created by randomly sampling 50% from these datasets. We performed random training-test set splitting for each dataset repeatedly to create 1,000 different training and test set pairs. For each dataset pair, the least absolute shrinkage and selection operator model was trained by five-fold cross-validation (CV) or nested CV with or without repetitions in the training set and tested with the test set, using the area under the curve (AUC) as an evaluation metric.Results: The AUCs in CV and testing varied widely based on data composition, especially with the undersampled datasets and the difficult task. The mean (±standard deviation) AUC difference between CV and testing was 0.029 (±0.022) for the simple task without undersampling and 0.108 (±0.079) for the difficult task with undersampling. In a training-test set pair, the AUC was high in CV but much lower in testing (0.840 and 0.650, respectively); in another dataset pair with the same task, however, the AUC was low in CV but much higher in testing (0.702 and 0.836, respectively). None of the CV methods helped overcome this issue.Conclusions: Machine learning after a single random training-test set split may lead to unreliable results in radiomics studies, especially when the sample size is small.

show abstract

“…The rst task was a relatively 'simple' task of differentiating between glioblastoma (GBM) and single metastasis, with reported accuracies of up to 89%. 11,12 The rst dataset consisted of 167 adult patients with a single GBM (n=109) or brain metastasis (n=58) that were pathologically con rmed following brain MRI from January 2014 to December 2017. The second task was a 'di cult' task of differentiating between low-vs. high-grade meningioma, with reported accuracies of less than 76% by conventional MRI.…”

Section: Subjectsmentioning

confidence: 99%

“…Both the datasets were from the same tertiary academic hospital, and some subsets of these patients were used in our previous reports. 11,15 MRI acquisition, image preprocessing, and radiomics feature extraction are described in Supplementary appendix.…”

Section: Subjectsmentioning

confidence: 99%

Radiomics machine learning study with a small sample size: single random training-test set split may result in unreliable results

Park

Ahn

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Objective: To determine how the estimated performance of a machine learning model varies according to how a dataset is split into training and test sets using brain tumor radiomics data, under different conditions.Materials and Methods: Two binary tasks with different levels of difficulty ('simple’ task, glioblastoma [GBM, n=109] vs. brain metastasis [n=58]; 'difficult’ task, low- [n=163] vs. high grade [n=95] meningiomas) were performed using radiomics features from magnetic resonance imaging (MRI). For each trial of the 1,000 different training-test set splits with a ratio of 7:3, a least absolute shrinkage and selection operator (LASSO) model was trained by 5-fold cross-validation (CV) in the training set and tested in the test set. The model stability and performance was evaluated according to the number of input features (from 1 to 50), the sample size (full vs. undersampled), and the level of difficulty. In addition to 5-fold CV without a repetition, three other CV methods were compared: 5-fold CV with 100 repetitions, nested CV, and nested CV with 100 repetitions.Results: The highest mean cross-validated area under the receiver operating characteristics curve (AUC) and the higher stability (lower AUC differences between training and testing) was achieved with 6 and 13 features from the GBM and meningioma task, respectively. For the simple task, simple task with undersampling, difficult task, and difficult task with undersampling, average mean AUCs were 0.947, 0.923, 0.795, and 0.764, and average AUC differences between training and testing were 0.029, 0.054, 0.053, and 0.108, respectively. Among four CV models, the most conservative method (i.e., lowest AUC and highest relative standard deviation [RSD]) was nested CV with 100 repetitions.Conclusions: A single random split of a dataset into training and test sets may lead to an unreliable report of model performance in radiomics machine learning studies, and reporting the mean and standard deviation of model performance metrics by performing nested and/or repeated CV on the entire dataset is suggested.

show abstract

Robust performance of deep learning for distinguishing glioblastoma from single brain metastasis using radiomic features: model development and validation

Cited by 75 publications

References 43 publications

Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential

Survival prediction of glioblastoma patients—are we there yet? A systematic review of prognostic modeling for glioblastoma and its clinical potential

Radiomics machine learning study with small sample size: single random training-test set split may result in unreliable results

Radiomics machine learning study with a small sample size: single random training-test set split may result in unreliable results

Contact Info

Product

Resources

About