SC-GAN: 3D self-attention conditional GAN with spectral normalization for multi-modal neuroimaging synthesis

Abstract

Image synthesis is one of the key applications of deep learning in neuroimaging, which enables shortening of the scan time and/or improve image quality; therefore, reducing the imaging cost and improving patient experience. Given the multi-modal and large-scale nature of neuroimaging data, the synthesis task is computationally challenging. 2D image synthesis networks do not take advantage of multi-dimensional spatial information and the 3D implementation has dimensionality problem, negatively affecting the network reliability. These limitations hinder the research and clinical applicability of deep learning-based neuroimaging synthesis. In this paper, we proposed a new network that is designed and optimized for the application of multi-modal 3D synthesis of neuroimaging data. The network is based on 3D conditional generative adversarial network (GAN), and employs spectral normalization and feature matching to stabilize the training process and ensure optimization convergence. We also added a self-attention module to model relationships between widely separated voxels. The performance of the network was evaluated by predicting positron emission tomography (PET) images, Fractional anisotropy (FA) and mean diffusivity (MD) maps from multi-modal magnetic resonance images (MRI) of 265 and 497 individuals correspondingly. The proposed network, called self-attention conditional GAN (SC-GAN), significantly outperformed conventional 2D conditional GAN and the 3D implementation, enabling robust 3D deep learning-based neuroimaging synthesis.

NiftyTorch: A Deep Learning framework for NeuroImaging

“…11) SC-GAN: In order to fully support neuroimage synthesis tasks, NiftyTorch incorporates multi-modality neuroimaging synthesis algorithm SC-GAN [14].…”

Manifold-aware Synthesis of High-resolution Diffusion from Structural Imaging

“…They demonstrate that structural images share sufficient information with the diffusion anisotropy of tissues to synthesize plausible 2D FA and MD slices. Similarly, in (Lan et al, 2020), a Self-attention Conditional GAN (SC-GAN) is used to generate FA and MD maps from different input modalities including structural T1w images. Their results indicate that both the 3D contextual information and the adversarial objective are important building blocks for the synthesis of diffusion data.…”

“…This body of work demonstrates the potential of generative models for the structural-to-diffusion synthesis of imaging data. Nevertheless, they remain limited (Gu et al, 2019;Lan et al, 2020;Son et al, 2019) in not exploiting the high-resolution information contained in the structural images to their full extent by either considering downsampled version of the T1w inputs or 2D slices with limited context. In addition, even though diffusion scalar maps are clinically useful, they mostly ignore fiber orientations and are of limited interest for tasks such as tractography or connectome visualization.…”

“…By constraining the generated diffusion to lie on the Riemannian manifold of 3 × 3 symmetric positive definite matrices (S 3 ++ ) for DT and on the n-Sphere S n manifold for ODF, we guarantee the mathematical coherence of the solution. As opposed to existing deep generative models that focused on the generation of diffusion derived scalar maps (Li and Wand, 2016;Gu et al, 2019;Lan et al, 2020), our model outputs complete diffusion schemes, here the 3 × 3 DT and the spherical harmonic coefficients of ODF. This important difference allows one to perform tractography, tractogram visualization and fiber bundle segmentation in addition to scalar maps computation directly from our network output while only requiring a T1w image as input.…”

Realistic Ultrasound Image Synthesis for Improved Classification of Liver Disease

“…The normalization is applied in the generator and discriminator simultaneously. Most recently SN was incorporated into GAN for improving low dose chest X-ray image resolution [26] and multi-modal neuroimage synthesis [13]. Self-Attention (SA) Module: SA module calculates the attention value between local pixel regions and helps to model global correlation in a wider range.…”