Multimodal MRI synthesis using unified generative adversarial networks

Abstract

Purpose Complementary information obtained from multiple contrasts of tissue facilitates physicians assessing, diagnosing and planning treatment of a variety of diseases. However, acquiring multiple contrasts magnetic resonance images (MRI) for every patient using multiple pulse sequences is time‐consuming and expensive, where, medical image synthesis has been demonstrated as an effective alternative. The purpose of this study is to develop a unified framework for multimodal MR image synthesis. Methods A unified generative adversarial network consisting of only a single generator and a single discriminator was developed to learn the mappings among images of four different modalities. The generator took an image and its modality label as inputs and learned to synthesize the image in the target modality, while the discriminator was trained to distinguish between real and synthesized images and classify them to their corresponding modalities. The network was trained and tested using multimodal brain MRI consisting of four different contrasts which are T1‐weighted (T1), T1‐weighted and contrast‐enhanced (T1c), T2‐weighted (T2), and fluid‐attenuated inversion recovery (Flair). Quantitative assessments of our proposed method were made through computing normalized mean absolute error (NMAE), peak signal‐to‐noise ratio (PSNR), structural similarity index measurement (SSIM), visual information fidelity (VIF), and naturalness image quality evaluator (NIQE). Results The proposed model was trained and tested on a cohort of 274 glioma patients with well‐aligned multi‐types of MRI scans. After the model was trained, tests were conducted by using each of T1, T1c, T2, Flair as a single input modality to generate its respective rest modalities. Our proposed method shows high accuracy and robustness for image synthesis with arbitrary MRI modality that is available in the database as input. For example, with T1 as input modality, the NMAEs for the generated T1c, T2, Flair respectively are 0.034 ± 0.005, 0.041 ± 0.006, and 0.041 ± 0.006, the PSNRs respectively are 32.353 ± 2.525 dB, 30.016 ± 2.577 dB, and 29.091 ± 2.795 dB, the SSIMs are 0.974 ± 0.059, 0.969 ± 0.059, and 0.959 ± 0.059, the VIF are 0.750 ± 0.087, 0.706 ± 0.097, and 0.654 ± 0.062, and NIQE are 1.396 ± 0.401, 1.511 ± 0.460, and 1.259 ± 0.358, respectively. Conclusions We proposed a novel multimodal MR image synthesis method based on a unified generative adversarial network. The network takes an image and its modality label as inputs and synthesizes multimodal images in a single forward pass. The results demonstrate that the proposed method is able to accurately synthesize multimodal MR images from a single MR image.

Generative Adversarial Network for Image Synthesis

“…This leads to superior quality of translated images compared to traditional Cycle-GAN models as well as the novel capability of flexibly translating an input image to any desired target domain. StarGAN was previously used in computer vision tasks such as a facial attribute transfer and a facial expression synthesis tasks [13], and was recently used for multimodal MRI synthesis [14].…”

Self‐supervised learning for accelerated 3D high‐resolution ultrasound imaging

“…Inspired by the tremendous success of deep learning in computer vision, 24–27 deep learning‐based methods have been recently investigated for medical image reconstruction, 28–31 analysis, 32–36 and synthesis 37,38 . Studies have demonstrated deep learning‐based approaches significantly outperform over CS‐based methods for image reconstruction 39‐42 .…”

“…[20][21][22][23] Inspired by the tremendous success of deep learning in computer vision, [24][25][26][27] deep learning-based methods have been recently investigated for medical image reconstruction, [28][29][30][31] analysis, [32][33][34][35][36] and synthesis. 37,38 Studies have demonstrated deep learning-based approaches significantly outperform over CS-based methods for image reconstruction. [39][40][41][42] Deep learning-based US image reconstruction algorithms and US beamforming methods have been proposed for processing both fully sampled and subsampled US RF data.…”

Non-isomorphic Inter-modality Graph Alignment and Synthesis for Holistic Brain Mapping

“…These methods either synthesize one modality from another (i.e., cross-modality) or map both modalities to a commonly shared domain. Specifically, generative adversarial networks (GANs) have held great promise in predicting medical images of different brain image modalities from a given modality [5,6,7]. For instance, [5] suggested a joint neuroimage synthesis and representation learning (JSRL) framework with transfer learning for subjective cognitive decline conversion prediction where they imputed missing PET images using MRI scans.…”

“…For instance, [5] suggested a joint neuroimage synthesis and representation learning (JSRL) framework with transfer learning for subjective cognitive decline conversion prediction where they imputed missing PET images using MRI scans. In addition, [6] proposed a unified GAN to train only a single generator and a single discriminator to learn the mappings among images of four different modalities. Furthermore, [7] translated a T1-weighted magnetic resonance imaging (MRI) to T2-weighted MRI using GAN.…”