Prospective Evaluation of Repeatability and Robustness of Radiomic Descriptors in Healthy Brain Tissue Regions In Vivo Across Systematic Variations in T2‐Weighted Magnetic Resonance Imaging Acquisition Parameters

Abstract

Background Radiomic descriptors from magnetic resonance imaging (MRI) are promising for disease diagnosis and characterization but may be sensitive to differences in imaging parameters. Objective To evaluate the repeatability and robustness of radiomic descriptors within healthy brain tissue regions on prospectively acquired MRI scans; in a test–retest setting, under controlled systematic variations of MRI acquisition parameters, and after postprocessing. Study Type Prospective. Subjects Fifteen healthy participants. Field Strength/Sequence A 3.0 T, axial T2‐weighted 2D turbo spin‐echo pulse sequence, 181 scans acquired (2 test/retest reference scans and 12 with systematic variations in contrast weighting, resolution, and acceleration per participant; removing scans with artifacts). Assessment One hundred and forty‐six radiomic descriptors were extracted from a contiguous 2D region of white matter in each scan, before and after postprocessing. Statistical Tests Repeatability was assessed in a test/retest setting and between manual and automated annotations for the reference scan. Robustness was evaluated between the reference scan and each group of variant scans (contrast weighting, resolution, and acceleration). Both repeatability and robustness were quantified as the proportion of radiomic descriptors that fell into distinct ranges of the concordance correlation coefficient (CCC): excellent (CCC > 0.85), good (0.7 ≤ CCC ≤ 0.85), moderate (0.5 ≤ CCC < 0.7), and poor (CCC < 0.5); for unprocessed and postprocessed scans separately. Results Good to excellent repeatability was observed for 52% of radiomic descriptors between test/retest scans and 48% of descriptors between automated vs. manual annotations, respectively. Contrast weighting (TR/TE) changes were associated with the largest proportion of highly robust radiomic descriptors (21%, after processing). Image resolution changes resulted in the largest proportion of poorly robust radiomic descriptors (97%, before postprocessing). Postprocessing of images with only resolution/acceleration differences resulted in 73% of radiomic descriptors showing poor robustness. Data Conclusions Many radiomic descriptors appear to be nonrobust across variations in MR contrast weighting, resolution, and acceleration, as well in test–retest settings, depending on feature formulation and postprocessing. Evidence Level 2 Technical Efficacy Stage 2

Test–Retest Data for the Assessment of Breast MRI Radiomic Feature Repeatability

“…To date, MRI test-retest studies for the evaluation of repeatable and reproducible features, have been conducted through phantom research 15,28,[31][32][33] and by the use of MRI exams of healthy volunteers or cancer patients. 17,19,20,32,[34][35][36] None of these studies investigated feature repeatability and/or reproducibility in human breast MRI exams, and only one study investigated a breast phantom. 28 The study of Saint Martin et al 28 showed the necessity of image preprocessing dedicated to breast MRI exams before using features in further analysis.…”

Challenges in ensuring the generalizability of image quantitation methods for MRI

“…Only specific subsets of radiomics features have been shown to be repeatable across readers 39,79,80 or implementations 71,81,82 . Parallel efforts in evaluating the reproducibility of radiomic features across controlled changes in echo time and repetition time in both phantom and in vivo settings also suggest that only a small subset of radiomic features are truly resilient to variations in acquisition settings 39,40,45,46,53 . Other than sources of image noise, a major factor impacting reproducibility and model generalizability is variations in image resolution, 83 where significant differences can reduce reproducible features to near zero 84 .…”

“…In radiomics, signal intensity values from contrast‐weighted MRI sequences (e.g., T 1 ‐weighted, T 2 ‐weighted, FLAIR, etc.) are often used to derive the texture features (Figure 4), but wide variations in acquisition settings such as echo/repetition times and flip angles between institutions, scanners, or body regions make a comparison of features derived from signal intensity values difficult or possibly even meaningless 39,40,43,45,53 . To address the variations in MR images, workflows therefore often include a variety of post‐processing steps 21,85,86 .…”

“…Used with permission from Alvarez-Jimenez et al 103 texture features (Figure 4), but wide variations in acquisition settings such as echo/repetition times and flip angles between institutions, scanners, or body regions make a comparison of features derived from signal intensity values difficult or possibly even meaningless. 39,40,43,45,53 To address the variations in MR images, workflows therefore often include a variety of post-processing steps. 21,85,86 However, the order of post-processing operations and how they are implemented can substantially impact the reproducibility of downstream radiomic features.…”

“…As a result, their grey‐scale properties are not necessarily a function of magnetic moments of water protons determined by hardware and software 51 . The variety of MRI systems and reconstruction algorithms across vendors makes enforcement of specific scanner hardware and software impractical, but radiomic descriptors and MR images are often not robust to these changes 52,53 . Generalizability and reduced variability may be achieved through protocol harmonization, 43,44 as well as following recommended protocols for specific hardware and software configurations 40,54 …”