Reproducibility in Radiomics: A Comparison of Feature Extraction Methods and Two Independent Datasets

Abstract

Radiomics involves the extraction of information from medical images not visible to the human eye. There is evidence these features can be used for treatment stratification and outcome prediction. However, there is much discussion about the reproducibility of results between different studies. This paper studies the reproducibility of CT texture features used in radiomics, comparing two feature extraction implementations namely Matlab toolkit and Pyradiomics when applied on independent datasets of CT scans of patients i) the open access RIDER dataset containing a set of repeat CT scans taken 15 minutes apart for 31 patients (RIDER Scan 1 and Scan 2 respectively) treated for lung cancer and ii) the open access HN1 dataset containing 137 patients treated for head and neck cancer. Gross tumor volume (GTV) manually outlined by an experienced observer available on both datasets was used. 43 common radiomics features available on Matlab and Pyradiomics were calculated using 2 intensity-level quantization methods with and without an intensity threshold. Cases were ranked for each feature for all combinations of quantization parameters and the Spearman’s rank coefficient, rs, calculated. Reproducibility was defined when a highly correlated feature in the RIDER dataset also correlated highly in the HN1 dataset and vice versa. 29 out of 43 reported stable features were found to be highly reproducible between Matlab and Pyradiomics implementations, having consistently high correlation in rank ordering for RIDER Scan 1 and RIDER Scan 2 (rs > 0.8). 18/43 reported features were common in RIDER and HN1 datasets, suggesting they may be agnostic to disease site. Useful radiomics features should be selected based on reproducibility. This study identified a set of features that meet this requirement and validated the methodology for evaluating reproducibility between datasets.