Reproducible radiomics through automated machine learning validated on twelve clinical applications

Abstract

Radiomics uses quantitative medical imaging features to predict clinical outcomes. While many radiomics methods have been described in the literature, these are generally designed for a single application. The aim of this study is to generalize radiomics across applications by proposing a framework to automatically construct and optimize the radiomics workflow per application. To this end, we formulate radiomics as a modular workflow, consisting of several components: image and segmentation preprocessing, feature extraction, feature and sample preprocessing, and machine learning. For each component, a collection of common algorithms is included. To optimize the workflow per application, we employ automated machine learning using a random search and ensembling. We evaluate our method in twelve different clinical applications, resulting in the following area under the curves: 1) liposarcoma (0.83); 2) desmoid-type fibromatosis (0.82); 3) primary liver tumors (0.81); 4) gastrointestinal stromal tumors (0.77); 5) colorectal liver metastases (0.68); 6) melanoma metastases (0.51); 7) hepatocellular carcinoma (0.75); 8) mesenteric fibrosis (0.81); 9) prostate cancer (0.72); 10) glioma (0.70); 11) Alzheimer's disease (0.87); and 12) head and neck cancer (0.84). Concluding, our method fully automatically constructs and optimizes the radiomics workflow, thereby streamlining the search for radiomics biomarkers in new applications. To facilitate reproducibility and future research, we publicly release six datasets, the software implementation of our framework (open-source), and the code to reproduce this study.

The effect of preprocessing filters on predictive performance in radiomics

“…In addition, not using explicit validation data might give rise to a positive bias, even though nested cross-validation should give a relatively unbiased estimation, if the external data follows the same distribution [13]. Comparing our AUC-ROCs to those of Starmans et al [28], it is striking that using the original set of features on four datasets (Desmoid, GIST, Lipo, Liver), the performance is slightly below the 95% CI reported there. For using all features, only in one case (Liver) was the performance lower, while no difference could be seen when using the best feature set.…”

Optimization of Preoperative Lymph Node Staging in Patients with Muscle-Invasive Bladder Cancer Using Radiomics on Computed Tomography

“…Using a single acquisition protocol could improve the performance unaffected by such variations, but it is not always feasible in a multicenter setting and limits the applicability. As the WORC method has previously successfully been used in similar settings [ 35 , 36 ], we do not expect that the poor performance can be explained by the variations in image acquisition alone.…”

“…The WORC radiomics method applied in this study has been previously validated in a variety of clinical applications [ 35 , 36 ]. In eleven of the twelve previous studies, the radiomics models had a better performance (mean AUCs between 0.68 and 0.94), and multiple features showed differences in univariate statistical testing, e.g., [ 39 , 44 , 45 , 46 ].…”

“…Therefore, further research is needed to investigate the use of various MRI protocols (e.g., T2-weighted, dynamic-contrast-enhanced and diffusion-weighted MRI [ 62 ],) for the detection of lymph nodes in patients with MIBC, as well as the relatively novel PET/MRI approach [ 63 ]. Since our radiomics approach has previously been proven successful for MRI [ 35 , 36 ], the same approach could be evaluated to differentiate pN+ and pN0 disease based on MRI.…”

“…The radiomics analysis were performed using the Workflow for Optimal Radiomics Classification (WORC) toolbox [ 35 , 36 ]; an overview of the radiomics methodology is depicted in Figure 1 . For each ROI (e.g., per lymph node or primary tumor segmentation), 564 features quantifying intensity, shape and texture were extracted from the CT scan.…”