Multi-Stage Harmonization for Robust AI across Breast MR Databases

Abstract

Radiomic features extracted from medical images may demonstrate a batch effect when cases come from different sources. We investigated classification performance using training and independent test sets drawn from two sources using both pre-harmonization and post-harmonization features. In this retrospective study, a database of thirty-two radiomic features, extracted from DCE-MR images of breast lesions after fuzzy c-means segmentation, was collected. There were 944 unique lesions in Database A (208 benign lesions, 736 cancers) and 1986 unique lesions in Database B (481 benign lesions, 1505 cancers). The lesions from each database were divided by year of image acquisition into training and independent test sets, separately by database and in combination. ComBat batch harmonization was conducted on the combined training set to minimize the batch effect on eligible features by database. The empirical Bayes estimates from the feature harmonization were applied to the eligible features of the combined independent test set. The training sets (A, B, and combined) were then used in training linear discriminant analysis classifiers after stepwise feature selection. The classifiers were then run on the A, B, and combined independent test sets. Classification performance was compared using pre-harmonization features to post-harmonization features, including their corresponding feature selection, evaluated using the area under the receiver operating characteristic curve (AUC) as the figure of merit. Four out of five training and independent test scenarios demonstrated statistically equivalent classification performance when compared pre- and post-harmonization. These results demonstrate that translation of machine learning techniques with batch data harmonization can potentially yield generalizable models that maintain classification performance.

ComBat Harmonization for MRI Radiomics

“…Different variants of ComBat have been proposed for radiomic feature harmonization, 14,15,[26][27][28] of which 2 were evaluated in our study: the "standard" variant with empirical Bayes estimation (ComBat-B) that has been used in several MRI radiomics studies [29][30][31][32][33] ; and a variant without empirical Bayes estimation (ComBat-NB), 32,34 which has been used less frequently, 34 but which may preferable if the number of features is substantially smaller than the number of participants, or if standard ComBat does not fit the data well (see https://github.com/ Jfortin1/neuroCombat_Rpackage/). Although both variants improved tissue classification, regardless of the classifier used, ComBat-NB was overall superior in our data set, as evidenced by LDA results that reflecting linear data separability, as well as MLP-NN results for 2 radiomic feature categories (Table 1), reflecting more sophisticated, nonlinear data separability.…”

“…These findings highlight the dependency of perceived harmonization performance on the choice of classifier and may in part explain why some studies did not report improved classification performance following ComBat harmonization. 32 Contrary to binary classification, that is, separation of only 2 classes, such as benign and malignant lesions, 5,33 or prediction of locoregional spread or control, 17,18 treatment response or relapse at a given time-point, [19][20][21]30 for which ComBat has been successfully used, our use of 3 tissue types with visually similar signal intensity and homogeneity on MRI makes the classification task more complex. Classification was further made difficult by our choice of T1-weighted Dixon images for radiomic feature extraction, where signal intensities showed only minor visible differences between tissues of interest.…”

Performance metric curve analysis framework to assess impact of the decision variable threshold, disease prevalence, and dataset variability in two-class classification

“…5 demonstrates that when the target 95% specificity operating point was used for this dataset, while there is a very low number of type I errors (7 [5,9]), the uncertainty in the already high number of type II errors was substantial (140 [120,169]). In contrast, the 95% CI of the sensitivity was low in the decision threshold region for the target 95% sensitivity classifier, which resulted in high precision in type I and type II errors for this operating point (48 [41,55] and 18 [14,22], respectively).…”

“…The details of the clinical dataset, including image acquisition protocol and clinical characterization, have been previously published. 19 – 23 Images were separated into two subsets by year of acquisition; one subset of lesions imaged in the period of 2015–2016 served as the training set and the other subset of lesions imaged in the year 2017 served as the independent test set ( Table 1 ). This longitudinal separation in a training and test cohort mimics how CADx would be deployed in the clinic after development.…”

Machine learning for multi-parametric breast MRI: radiomics-based approaches for lesion classification

“…Hence, there is a strong need for feature harmonization, to allow consistent findings in radiomics multicenter studies (Da-Ano et al 2020). Recently, Whitney et al (Whitney et al 2021b) have proposed a batch harmonization approach for robust application of AI across different MR breast databases. Batch harmonization of radiomic features extracted from DCE images from two different databases was applied to a ML classification workflow.…”