CT Super-Resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE)

You, Chenyu; Li, Guang; Zhang, Yi; Zhang, Xiaoliu; Shan, Hongming; Ju, Shenghong; Zhen, Zhao; Zhang, Zhuiyang; Cong, Wenxiang; Vannier, Michael W.; Saha, Punam K.; Wang, Ge

doi:10.1109/tmi.2019.2922960

Cited by 398 publications

(292 citation statements)

References 83 publications

Supporting

Mentioning

291

Contrasting

Order By: Relevance

“…Since its introduction in 2014, generative adversarial network (GAN) has achieved remarkable success in generative image modeling and has shown outstanding performances in numerous applications . The architecture of the generative adversarial network integrates two competing networks, a generative network and a discriminative network, into one framework.…”

Section: Introductionmentioning

confidence: 99%

Automatic multiorgan segmentation in thorax CT images using U‐net‐GAN

et al. 2019

View full text Add to dashboard Cite

Purpose Accurate and timely organs‐at‐risk (OARs) segmentation is key to efficient and high‐quality radiation therapy planning. The purpose of this work is to develop a deep learning‐based method to automatically segment multiple thoracic OARs on chest computed tomography (CT) for radiotherapy treatment planning. Methods We propose an adversarial training strategy to train deep neural networks for the segmentation of multiple organs on thoracic CT images. The proposed design of adversarial networks, called U‐Net‐generative adversarial network (U‐Net‐GAN), jointly trains a set of U‐Nets as generators and fully convolutional networks (FCNs) as discriminators. Specifically, the generator, composed of U‐Net, produces an image segmentation map of multiple organs by an end‐to‐end mapping learned from CT image to multiorgan‐segmented OARs. The discriminator, structured as an FCN, discriminates between the ground truth and segmented OARs produced by the generator. The generator and discriminator compete against each other in an adversarial learning process to produce the optimal segmentation map of multiple organs. Our segmentation results were compared with manually segmented OARs (ground truth) for quantitative evaluations in geometric difference, as well as dosimetric performance by investigating the dose‐volume histogram in 20 stereotactic body radiation therapy (SBRT) lung plans. Results This segmentation technique was applied to delineate the left and right lungs, spinal cord, esophagus, and heart using 35 patients’ chest CTs. The averaged dice similarity coefficient for the above five OARs are 0.97, 0.97, 0.90, 0.75, and 0.87, respectively. The mean surface distance of the five OARs obtained with proposed method ranges between 0.4 and 1.5 mm on average among all 35 patients. The mean dose differences on the 20 SBRT lung plans are −0.001 to 0.155 Gy for the five OARs. Conclusion We have investigated a novel deep learning‐based approach with a GAN strategy to segment multiple OARs in the thorax using chest CT images and demonstrated its feasibility and reliability. This is a potentially valuable method for improving the efficiency of chest radiotherapy treatment planning.

show abstract

Section: Introductionmentioning

confidence: 99%

Automatic multiorgan segmentation in thorax CT images using U‐net‐GAN

et al. 2019

View full text Add to dashboard Cite

show abstract

“…The neural network architecture used in this study can be made more efficient with wide activation network trimming (Yu et al, ) or more flexible with unpaired, unsupervised cyclical networks (You et al, ). The choice of loss function weights will affect the interplay between generator and discriminator (features and textures), and the availability of training data and augmentation further adds avenues of algorithmic improvement.…”

Section: Discussionmentioning

confidence: 99%

“…Cycle‐consistent GANs (cycleGANs) (Zhu et al, ) have been used as a semisupervised method of generating SR images by training on unpaired SR and LR datasets (You et al, ). While SRGAN results in features that look to the human eye as realistic when surveyed, the resulting generated SR images are lower in pixelwise accuracy compared to SRCNN due to pixelwise mismatch (You et al, ). Since

normalμ

CT images contain significant amounts of image noise and texture as high‐frequency features, this is also inadvertently recovered.…”

Section: Introductionmentioning

confidence: 99%

Boosting Resolution and Recovering Texture of 2D and 3D Micro‐CT Images with Deep Learning

Wang

Armstrong

Mostaghimi

2020

Water Resources Research

View full text Add to dashboard Cite

Simulation of flow directly at the pore scale depends on high-quality digital rock images but is constrained by detector hardware. A trade-off between the image field of view (FOV) and image resolution is made. This can be compensated for with superresolution (SR) techniques that take a wide FOV, low-resolution (LR) image, and superresolve a high resolution (HR). The Enhanced Deep Super Resolution Generative Adversarial Network (EDSRGAN) is trained on the DeepRock-SR, a diverse compilation of raw and processed micro-computed tomography (μCT) images in 2D and 3D. The 2D and 3D trained networks show comparable performance of 50% to 70% reduction in relative error over bicubic interpolation with minimal computational cost during usage. Texture regeneration with EDSRGAN shows superior visual similarity versus Super Resolution Convolutional Neural Network (SRCNN) and other methods. Difference maps show SRCNN recovers large-scale edge features while EDSRGAN regenerates perceptually indistinguishable high-frequency texture. Physical accuracy is measured by permeability and phase topology on consistently segmented images, showing EDSRGAN results achieving the closest match. Performance is generalized with augmentation, showing high adaptability to noise and blur. HR images are fed into the network, generating HR-SR images to extrapolate network performance to subresolution features present in the HR images themselves. Underresolution features are regenerated despite operating outside of trained specifications. Comparison with scanning electron microscopy (SEM) images shows details are consistent with the underlying geometry. Images that are normally constrained by the mineralogy of the rock, by fast transient imaging, or by the energy of the source can be superresolved accurately for further analysis. Plain Language SummaryWhen capturing an X-ray image of the insides of a rock sample (or any opaque object), hardware limitations on the image quality and size exist. These limitations can be overcome with the use of machine learning algorithms that "superresolve" a lower resolution image. Once trained, the machine algorithm can sharpen otherwise blurry features and regenerate the underlying texture of the imaged object. We train such an algorithm on a large and wide array of digital rock images and test its flexibility on some images that it had never seen before, as well as on some very high quality images that it was not trained to superresolve. The results of training and testing the algorithm shows a promising degree of accuracy and flexibility in handling a wide array of images of different quality and allows for higher quality images to be generated for use in other image-based analysis techniques.

show abstract

“…Since PSNR and SSIM values cannot guarantee a visually favorable result, we employ an additional metric for the final results, namely the information fidelity criterion (IFC) [48]. Although IFC is scarcely used in literature [15], Yang et al [49] pointed out that IFC is correlated well with the human perception of SR images. As for PSNR and SSIM, higher IFC values indicate better results.…”

Section: B Experimental Setup 1) Evaluation Metricsmentioning

confidence: 99%

Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

Ionescu

Verga

2020

IEEE Access

View full text Add to dashboard Cite

Computed Tomography (CT) scanners that are commonly-used in hospitals and medical centers nowadays produce low-resolution images, e.g. one voxel in the image corresponds to at most onecubic millimeter of tissue. In order to accurately segment tumors and make treatment plans, radiologists and oncologists need CT scans of higher resolution. The same problem appears in Magnetic Resonance Imaging (MRI). In this paper, we propose an approach for the single-image super-resolution of 3D CT or MRI scans. Our method is based on deep convolutional neural networks (CNNs) composed of 10 convolutional layers and an intermediate upscaling layer that is placed after the first 6 convolutional layers. Our first CNN, which increases the resolution on two axes (width and height), is followed by a second CNN, which increases the resolution on the third axis (depth). Different from other methods, we compute the loss with respect to the ground-truth high-resolution image right after the upscaling layer, in addition to computing the loss after the last convolutional layer. The intermediate loss forces our network to produce a better output, closer to the ground-truth. A widely-used approach to obtain sharp results is to add Gaussian blur using a fixed standard deviation. In order to avoid overfitting to a fixed standard deviation, we apply Gaussian smoothing with various standard deviations, unlike other approaches. We evaluate the proposed method in the context of 2D and 3D super-resolution of CT and MRI scans from two databases, comparing it to related works from the literature and baselines based on various interpolation schemes, using 2× and 4× scaling factors. The empirical study shows that our approach attains superior results to all other methods. Moreover, our subjective image quality assessment by human observers reveals that both doctors and regular annotators chose our method in favor of Lanczos interpolation in 97.55% cases for an upscaling factor of 2× and in 96.69% cases for an upscaling factor of 4×. In order to allow others to reproduce our state-of-the-art results, we provide our code as open source at https://github.com/lilygeorgescu/3d-super-res-cnn. INDEX TERMSConvolutional neural networks, single-image super-resolution, CT images, MRI images, medical image super-resolution.

show abstract

CT Super-Resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE)

Cited by 398 publications

References 83 publications

Automatic multiorgan segmentation in thorax CT images using U‐net‐GAN

Automatic multiorgan segmentation in thorax CT images using U‐net‐GAN

Boosting Resolution and Recovering Texture of 2D and 3D Micro‐CT Images with Deep Learning

Convolutional Neural Networks With Intermediate Loss for 3D Super-Resolution of CT and MRI Scans

Contact Info

Product

Resources

About