Intra-Scanner and Inter-Scanner Reproducibility of Automatic White Matter Hyperintensities Quantification

Abstract

Objectives: To evaluate white matter hyperintensities (WMH) quantification reproducibility from multiple aspects of view and examine the effects of scan–rescan procedure, types of scanner, imaging protocols, scanner software upgrade, and automatic segmentation tools on WMH quantification results using magnetic resonance imaging (MRI). Methods: Six post-stroke subjects (4 males; mean age = 62.8, range = 58–72 years) were scanned and rescanned with both 3D T1-weighted, 2D and 3D T2-weighted fluid-attenuated inversion recovery (T2-FLAIR) MRI across four different MRI scanners within 12 h. Two automated WMH segmentation and quantification tools were used to measure WMH volume based on each MR scan. Robustness was assessed using the coefficient of variation (CV), Dice similarity coefficient (DSC), and intra-class correlation (ICC). Results: Experimental results show that the best reproducibility was achieved by using 3D T2-FLAIR MRI under intra-scanner setting with CV ranging from 2.69 to 2.97%, while the largest variability resulted from comparing WMH volumes measured based on 2D T2-FLAIR MRI with those of 3D T2-FLAIR MRI, with CV values in the range of 15.62%–29.33%. The WMH quantification variability based on 2D MRIs is larger than 3D MRIs due to their large slice thickness. The DSC of WMH segmentation labels between intra-scanner MRIs ranges from 0.63 to 0.77, while that for inter-scanner MRIs is in the range of 0.63–0.65. In addition to image acquisition, the choice of automatic WMH segmentation tool also has a large impact on WMH quantification. Conclusion: WMH reproducibility is one of the primary issues to be considered in multicenter and longitudinal studies. The study provides solid guidance in assisting multicenter and longitudinal study design to achieve meaningful results with enough power.

Integrating large-scale neuroimaging research datasets: harmonisation of white matter hyperintensity measurements across Whitehall and UK Biobank datasets

“…However, the scope of this evaluation was to explore how differences in manual ratings can impact a supervised segmentation method like BIANCA. To help quantifying the variability caused by manual segmentation we looked at the average agreement (DI) range and found that our between- and within-rater agreement is comparable with the scan-rescan agreement in WMHs assessed in a previous study (inter-scanner range: 0.63–0.65; intra-scanner range: 0.63–0.77) (Guo et al, 2019). This suggests that the impact of the rater on the final segmentation is comparable to the effect of repeating the acquisition using the same settings.…”

“…We then identified optimal pre-processing and analysis strategies to reduce non-biological variability across datasets, while retaining or taking into account (modelling) the biological variability. Effect of rater : in the training phase, BIANCA requires manually delineated WMH masks, which are known to suffer from inter- and intra-rater variability (Guo et al, 2019). We wanted to assess whether BIANCA trained with different manual masks (either multiple annotations by different raters or repeated annotations by the same rater) generates WMH segmentations that are more or less variable than the manual annotations among themselves.…”

“…Fortin et al, 2018; Zanadifar et al, 2018) and diffusion MRI (e.g. Fortin et al, 2017; Mirzaalian et al, 2016), harmonisation approaches for T2-weighted scans and the quantification of WMHs (and other lesions) are still lacking, despite the recognition that biases are also present in this modality (Shinohara et al, 2017; Guo et al, 2019). Consortia and working groups (Wardlaw et al, 2013; Smith et al, 2019) recognised the need to standardise the assessment of SVD and proposed a set of standard definitions, acquisition protocols and a framework for developing neuroimaging biomarkers of cerebral small vessel disease.…”

“…This is because, even after careful protocol harmonisation, systematic differences in the images can remain (related to scanner vendor, model, non-linearity of imaging gradients, magnetic field homogeneity, signal-to-noise ratio etc.) leading to bias in the MRI-derived measures (Kruggel et al, 2010; Potvin et al, 2019; Shinohara et al, 2017; Mirzaalian et al, 2016; Guo et al, 2019). At the image pre-processing level, harmonisation strategies aim to remove the non-biological variability, leading to more similar images (Mirzaalian et al, 2016; Dewey et al, 2018), and to design/use analysis tools or pipelines that give consistent performance on different datasets and produce well-matched measures (Erus et al, 2018; Zandifar et al, 2018; Guo et al, 2019).…”

“…leading to bias in the MRI-derived measures (Kruggel et al, 2010; Potvin et al, 2019; Shinohara et al, 2017; Mirzaalian et al, 2016; Guo et al, 2019). At the image pre-processing level, harmonisation strategies aim to remove the non-biological variability, leading to more similar images (Mirzaalian et al, 2016; Dewey et al, 2018), and to design/use analysis tools or pipelines that give consistent performance on different datasets and produce well-matched measures (Erus et al, 2018; Zandifar et al, 2018; Guo et al, 2019). At the analysis level, harmonisation approaches aim to make measures consistent across datasets, some also modelling biological variability, so that characteristics of imaging site and study are removed from the data (Fortin et al, 2017; Fortin et al, 2018; Pomponio 2020).…”