Breaking the Dilemma of Medical Image-to-image Translation

Abstract

Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field of medical image-to-image translation. However, neither modes are ideal. The Pix2Pix mode has excellent performance. But it requires paired and well pixel-wise aligned images, which may not always be achievable due to respiratory motion or anatomy change between times that paired images are acquired. The Cycle-consistency mode is less stringent with training data and works well on unpaired or misaligned images. But its performance may not be optimal. In order to break the dilemma of the existing modes, we propose a new unsupervised mode called RegGAN for medical image-to-image translation. It is based on the theory of "loss-correction". In RegGAN, the misaligned target images are considered as noisy labels and the generator is trained with an additional registration network to fit the misaligned noise distribution adaptively. The goal is to search for the common optimal solution to both image-to-image translation and registration tasks. We incorporated RegGAN into a few state-of-the-art image-to-image translation methods and demonstrated that RegGAN could be easily combined with these methods to improve their performances. Such as a simple CycleGAN in our mode surpasses latest NICEGAN even though using less network parameters. Based on our results, RegGAN outperformed both Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired data. RegGAN is insensitive to noises which makes it a better choice for a wide range of scenarios, especially for medical image-to-image translation tasks in which well pixel-wise aligned data are not available. Code and data used in this study can be found at https://github.com/Kid-Liet/Reg-GAN.

CBCT-to-CT Translation Using Registration-Based Generative Adversarial Networks in Patients with Head and Neck Cancer

“…As mentioned previously, the registration network (RegNet) is based on the work of Kong et al [ 35 ]. The network is trained to acquire prior knowledge regarding the distribution of misaligned noise such that it can dynamically predict the deformation vector field (DVF), which in turn can be used to estimate the correct label, which is unknown in the real world.…”

“…To preserve the HU values between the source and target images, a loss function for the generator is implemented using loss and MSE loss: where as in the original implementation [ 39 ]. Because RegNet is added during training after the generator to account for misaligned targets, the total loss function is defined as where and are the weights for registration loss and smoothing loss, respectively, as suggested by Kong et al [ 35 ].…”

“…To evaluate the quality of the generated images, we used commonly used metrics, such as mean absolute error (MAE), root mean square error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM), to compare them with the ground truth images, which are also two-dimensional. We selected these metrics based on a review of similar studies in the literature [ 27 , 29 , 33 , 35 , 41 ], which enables our results to be compared with previous research.…”

“…Arar et al [ 34 ] were the first to demonstrate performance improvements in GANs for cross-modality image translation by adding a RegNet to either the input or output of the GAN, allowing the network to learn both translation and registration simultaneously. Kong et al [ 35 ] introduced a RegNet to GANs to enable them to learn the misaligned noise distribution. This allowed the network to dynamically correct the noisy labels during training with the goal of finding a joint optimal solution for both image-to-image translation and image registration.…”

“…They demonstrated the effectiveness of this technique in MRI T1-T2 image translation. Yang et al [ 36 ] investigated MRI cross-modality image translation, similar to Kong et al [ 35 ], by incorporating a deformable registration method into GANs. Chen et al [ 37 ] exploited a registration network to generate a deformation field that helps in improving GAN performance for multi-modal CR-MRI image translation.…”