Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network

Abstract

Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image registration and as supplementary training samples to boost the generalization capability of a classifier. Especially, synthesized computed tomography (CT) data can provide Xray attenuation map for radiation therapy planning. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 3D images using unpaired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distortion in cross-modality synthesis and 3) improving volume segmentation by using synthetic data for modalities with limited training samples. We show that these goals can be achieved with an end-to-end 3D convolutional neural network (CNN) composed of mutuallybeneficial generators and segmentors for image synthesis and segmentation tasks. The generators are trained with an adversarial loss, a cycle-consistency loss, and also a shapeconsistency loss, which is supervised by segmentors, to reduce the geometric distortion. From the segmentation view, the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors prompt each other alternatively in an end-to-end training fashion. With extensive experiments on a dataset including a total of 4,496 CT and magnetic resonance imaging (MRI) cardiovascular volumes, we show both tasks are beneficial to each other and coupling these two tasks results in better performance than solving them exclusively.

Cardiac substructure segmentation with deep learning for improved cardiac sparing

“…Data quantity may also be increased through utilizing unlabeled images for unsupervised learning via generative models such as a cycle or a stacked GAN, which implement multiple GANs for data synthesis. Recently, Zhang et al proposed a novel cardiac chamber segmentation method using a GAN integrating cycle and shape consistency. They obtained DSCs comparable to atlas segmentations (DSC ~ 0.75) on CT and MRI by using ~14% real data and augmenting their dataset by incorporating synthetic MRI and CT data into training.…”

Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation

“…For cross-modality adaptation, Jiang et al [36] first transform CT images to resemble MRI appearance using CycleGAN with tumor-aware loss, then the generated MRI images are combined with a few real MRI data for semi-supervised tumor segmentation. In [37] and [38], CycleGAN is combined with a segmentation network to compose an end-to-end framework. Compared to [37] and [38], a key characteristic of our approach is the shared encoder for both image transformation and segmentation task.…”

A novel MRI segmentation method using CNN‐based correction network for MRI‐guided adaptive radiotherapy

“…Generative Adversarial Networks (GAN) have also been applied to help CNN to predict plausible structural labeling by employing a discriminator which learns to discriminate between the CNN‐predicted labels and the ground truth labels . One disadvantage of GAN is that it is very difficult to train and is very sensitive to hyperparameters .…”