Reproducibility analysis of multi‐institutional paired expert annotations and radiomic features of the Ivy Glioblastoma Atlas Project (Ivy GAP) dataset

Abstract

Purpose The availability of radiographic magnetic resonance imaging (MRI) scans for the Ivy Glioblastoma Atlas Project (Ivy GAP) has opened up opportunities for development of radiomic markers for prognostic/predictive applications in glioblastoma (GBM). In this work, we address two critical challenges with regard to developing robust radiomic approaches: (a) the lack of availability of reliable segmentation labels for glioblastoma tumor sub‐compartments (i.e., enhancing tumor, non‐enhancing tumor core, peritumoral edematous/infiltrated tissue) and (b) identifying “reproducible” radiomic features that are robust to segmentation variability across readers/sites. Acquisition and validation methods From TCIA’s Ivy GAP cohort, we obtained a paired set (n = 31) of expert annotations approved by two board‐certified neuroradiologists at the Hospital of the University of Pennsylvania (UPenn) and at Case Western Reserve University (CWRU). For these studies, we performed a reproducibility study that assessed the variability in (a) segmentation labels and (b) radiomic features, between these paired annotations. The radiomic variability was assessed on a comprehensive panel of 11 700 radiomic features including intensity, volumetric, morphologic, histogram‐based, and textural parameters, extracted for each of the paired sets of annotations. Our results demonstrated (a) a high level of inter‐rater agreement (median value of DICE ≥0.8 for all sub‐compartments), and (b) ≈24% of the extracted radiomic features being highly correlated (based on Spearman’s rank correlation coefficient) to annotation variations. These robust features largely belonged to morphology (describing shape characteristics), intensity (capturing intensity profile statistics), and COLLAGE (capturing heterogeneity in gradient orientations) feature families. Data format and usage notes We make publicly available on TCIA’s Analysis Results Directory (https://doi.org/10.7937/9j41‐7d44), the complete set of (a) multi‐institutional expert annotations for the tumor sub‐compartments, (b) 11 700 radiomic features, and (c) the associated reproducibility meta‐analysis. Potential applications The annotations and the associated meta‐data for Ivy GAP are released with the purpose of enabling researchers toward developing image‐based biomarkers for prognostic/predictive applications in GBM.

Prospective Evaluation of Repeatability and Robustness of Radiomic Descriptors in Healthy Brain Tissue Regions In Vivo Across Systematic Variations in T2‐Weighted Magnetic Resonance Imaging Acquisition Parameters

“…Among feature families, co-occurrence based Haralick and COLLAGE descriptors consistently showed good to excellent repeatability performance, which was only marginally changed after postprocessing; in-line with previous findings across a number of different organs. 9,10,32,33 In contrast, gradient and Laws descriptors were poorly repeatable in both comparisons (both before and after postprocessing) which resonates with studies suggesting their sensitivity to even marginal imaging or annotation differences. 34,35 The difference in repeatability performance between edge-based and co-occurrence descriptors further suggests that first-order derivatives (used in Laws and gradient operators) may be more sensitive than higher-order derivatives (used in Haralick and COLLAGE).…”

“…Second, when comparing radiomic descriptors between manually and automatically generated WM annotations on the same reference images, approximately 40% of descriptors showed poor repeatability and nearly half showed good to excellent repeatability. Radiomic descriptors thus exhibited poorer repeatability between manual and automated annotations than between test/retest scans, as has been observed previously 31,32 and potentially due to the moderate overlap between the two sets of annotations. Among feature families, co-occurrence based Haralick and COLLAGE descriptors consistently showed good to excellent repeatability performance, which was only marginally changed after postprocessing; in-line with previous findings across a number of different organs.…”

World Cancer Day 2021 - Perspectives in Pediatric and Adult Neuro-Oncology

“…Furthermore, radiomics and the application of machine learning/artificial intelligence to diagnostic MRI scans has the potential to identify early tumor recurrence/progression, distinguish pseudoprogression from progression (42,43) as well as to identify imaging signatures that are relatively specific to molecular subgroups of the more common diagnoses in adults (GBM, oligodendroglial tumors, low grade gliomas) (44,45) and children (low grade gliomas, medulloblastoma, ependymoma, diffuse midline gliomas) (46,47). While several techniques have been described, none have achieved widespread clinical acceptance for routine use.…”

Fully Automated MR Based Virtual Biopsy of Cerebral Gliomas

“…All approaches come with their own advantages and disadvantages. While the traditional approach with radiomics has traceable features, allowing them to be replicated in studies [ 13 ], the segmentation required and the often poorer prediction compared to CNN approaches is a major point of criticism [ 5 ]. In contrast, feature extraction by means of CNN often delivers better results [ 5 ], but these are less comprehensible and cannot be compared in studies to the same extent; simultaneously, the often manual or semi-automatic segmentation is a major point of criticism here as well.…”