On the proper use of structural similarity for the robust evaluation of medical image synthesis models

Abstract

To propose good practices for using the structural similarity metric (SSIM) and reporting its value. SSIM is one of the most popular image quality metrics in use in the medical image synthesis community because of its alleged superiority over voxel-by-voxel measurements like the average error or the peak signal noise ratio (PSNR). It has seen massive adoption since its introduction, but its limitations are often overlooked. Notably, SSIM is designed to work on a strictly positive intensity scale, which is generally not the case in medical imaging. Common intensity scales such as the Houndsfield units (HU) contain negative numbers, and they can also be introduced by image normalization techniques such as the z-normalization. Methods: We created a series of experiments to quantify the impact of negative values in the SSIM computation. Specifically, we trained a three-dimensional (3D) U-Net to synthesize T2-weighted MRI from T1-weighted MRI using the BRATS 2018 dataset. SSIM was computed on the synthetic images with a shifted dynamic range. Next, to evaluate the suitability of SSIM as a loss function on images with negative values, it was used as a loss function to synthesize znormalized images. Finally, the difference between two-dimensional (2D) SSIM and 3D SSIM was investigated using multiple 2D U-Nets trained on different planes of the images. Results:The impact of the misuse of the SSIM was quantified; it was established that it introduces a large downward bias in the computed SSIM. It also introduces a small random error that can change the relative ranking of models. The exact values for this bias and error depend on the quality and the intensity histogram of the synthetic images. Although small, the reported error is significant considering the small SSIM difference between state-of -the-art models. It was shown therefore that SSIM cannot be used as a loss function when images contain negative values due to major errors in the gradient calculation, resulting in under-performing models. 2D SSIM was also found to be overestimated in 2D image synthesis models when computed along the plane of synthesis, due to the discontinuities between slices that is typical of 2D synthesis methods. Conclusion: Various types of misuse of the SSIM were identified, and their impact was quantified. Based on the findings, this paper proposes good practices when using SSIM, such as reporting the average over the volume of the image containing tissue and appropriately defining the dynamic range.

Deep learning methods for enhancing cone‐beam CT image quality toward adaptive radiation therapy: A systematic review

“…Furthermore, reporting uncorrected CBCT image quality is necessary to gauge the relative improvement. Similar recommendations were recently echoed for SSIM: The authors recommended reporting the average SSIM within the patient body contour and ensuring that an appropriate dynamic range is used 118 . As previously discussed, the FID is a popular image quality metric used in assessing generative models that lack a ground truth 94 .…”

“…Similar recommendations were recently echoed for SSIM: The authors recommended reporting the average SSIM within the patient body contour and ensuring that an appropriate dynamic range is used. 118 As previously discussed, the FID is a popular image quality metric used in assessing generative models that lack a ground truth. 94 The distance metric compares the distribution of deep features between two datasets irrespective of their pixel-wise alignment, thereby addressing the inherent flaw of using MAE.…”

Deep learning prediction of post‐SBRT liver function changes and NTCP modeling in hepatocellular carcinoma based on DGAE‐MRI

“…45 The maximum achievable value is 1., if and only if two images are identical. SSIM has been used for evaluating image processing results in different medical applications, such as image synthesis, 46 artifact suppression, 47 predicting post-contrast MR images from pre-contrast images, 48 predicting PET images from MRIs, 49 image generation using GANs. 50,51 The range of SSIM for these applications are wide (∼0.2 to ∼0.99).…”

Accelerated respiratory‐resolved 4D‐MRI with separable spatio‐temporal neural networks

“…The image quality was measured by the average SSIM and the normalized root‐mean‐square error (NRMSE) over the respiratory phases between the model reconstruction and the XD‐GRASP reconstruction. The NRMSE was computed as $$\operatorname{NRMSE}(I_{\rm est}, I_{\rm target}) = \sqrt {1/M \sum (I_{\rm est}- I_{\rm target})^2} / \overline{\vert I_{\rm target}\vert }$$, where M is the number of voxels and $$\overline{\vert I_{\rm target}\vert }$$ is the mean absolute value of I target within the anatomy 57 . The motion estimation quality was quantified in two ways: …”

“…The NRMSE was computed as NRMSE(I est , I target ) = √ 1∕M ∑ (I est − I target ) 2 ∕|I target |, where M is the number of voxels and |I target | is the mean absolute value of I target within the anatomy. 57 The motion estimation quality was quantified in two ways:…”