3D Auto-Context-Based Locality Adaptive Multi-Modality GANs for PET Synthesis

Abstract

Positron emission tomography (PET) has been substantially used recently. To minimize the potential health risk caused by the tracer radiation inherent to PET scans, it is of great interest to synthesize the high-quality PET image from the low-dose one to reduce the radiation exposure. In this paper, we propose a 3D auto-context-based locality adaptive multi-modality generative adversarial networks model (LA-GANs) to synthesize the high-quality FDG PET image from the low-dose one with the accompanying MRI images that provide anatomical information. Our work has four contributions. First, different from the traditional methods that treat each image modality as an input channel and apply the same kernel to convolve the whole image, we argue that the contributions of different modalities could vary at different image locations, and therefore a unified kernel for a whole image is not optimal. To address this issue, we propose a locality adaptive strategy for multi-modality fusion. Second, we utilize 1×1×1 kernel to learn this locality adaptive fusion so that the number of additional parameters incurred by our method is kept minimum. Third, the proposed locality adaptive fusion mechanism is learned jointly with the PET image synthesis in a 3D conditional GANs model, which generates high-quality PET images by employing large-sized image patches and hierarchical features. Fourth, we apply the auto-context strategy to our scheme and propose an auto-context LA-GANs model to further refine the quality of synthesized images. Experimental results show that our method outperforms the traditional multi-modality fusion methods used in deep networks, as well as the state-of-the-art PET estimation approaches.

Hi-Net: Hybrid-Fusion Network for Multi-Modal MR Image Synthesis

“…Wolterink et al [18] utilized GAN to transform low-dose CT into routine-dose CT images. Wang et al [16] also demonstrated promising results when using a GAN to estimate high-dose PET images from low-dose ones. The list of GAN-based methods proposed for medical synthesis is extensive [2], [19], [38]- [40].…”

Application and Construction of Deep Learning Networks in Medical Imaging

“…Xiang et al input real low-dose PET (25% dose) and T1 MRI images into a deep autocontext CNN network to predict standard-dose PET images [124]. Similarly, Wang et al utilized a locality adaptive multimodality generative adversarial networks (LA-GANs) to also synthesize high-quality PET images from low-dose PET and T1 MRI images alone [125] or combined with DTI [126]. Meanwhile, Chen et al utilized simultaneously acquired MR images and simulated ultra-low-dose PET images to synthesize full-dose amyloid PET images using an encoder-decoder CNN [127].…”

High‐performance dynamic magnetic resonance image reconstruction and synthesis employing deep feature learning convolutional networks

“…The highlights created by profound learning mirror the human observation through different activities, for example, convolution and pooling, and this makes better component descriptors contrasted with the low-level model. 12 In CNN, the previous architectures, AlexNet, GoogLeNet, and VGG, are some accurate architecture. Among these, AlexNet model has improved inception module, requires less memory utilization and simple calculation, and it is therefore more famous than others.…”