StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation

Abstract

Figure 1. Multi-domain image-to-image translation results on the CelebA dataset via transferring knowledge learned from the RaFD dataset. The first and sixth columns show input images while the remaining columns are images generated by StarGAN. Note that the images are generated by a single generator network, and facial expression labels such as angry, happy, and fearful are from RaFD, not CelebA. AbstractRecent studies have shown remarkable success in imageto-image translation for two domains. However, existing approaches have limited scalability and robustness in handling more than two domains, since different models should be built independently for every pair of image domains. To address this limitation, we propose StarGAN, a novel and scalable approach that can perform image-to-image translations for multiple domains using only a single model. Such a unified model architecture of StarGAN allows simultaneous training of multiple datasets with different domains within a single network. This leads to StarGAN's superior quality of translated images compared to existing models as well as the novel capability of flexibly translating an input image to any desired target domain. We empirically demonstrate the effectiveness of our approach on a facial attribute transfer and a facial expression synthesis tasks.

Batch Equalization with a Generative Adversarial Network

“…After training the generator in the GAN framework with the discriminator, we can transfer the entire image dataset to the same batch target t using the generator model G such that the batch effect is equalized/mediated and downstream analysis can then be performed on this generated ideal dataset. (Choi et al, 2017). a) Training procedure for the discriminator D. The discriminator D takes an image (either real or generated) and predicts its source (real/generated), which batch it comes from, and biological features such as cell size.…”

“…Since our framework heavily relies on the original StarGAN (Choi et al, 2017), we will first do a brief review of the StarGAN formulation, and introduce our extension for representation disentanglement in the following section. Under the GAN framework, our discriminator D will learn to predict whether an image is a real image from the original data distribution or a generated image from G. This is the image source prediction D src (X).…”

“…If we treat each batch as an individual style, we can formulate the batch normalization problem as a batch equalization problem where we learn to transfer all images to the same batch with the biological (or semantic) feature intact such that the batch effect can be mediated naturally ( Figure 1c). In this work, we leverage StarGAN (Choi et al, 2017), a multi-domain/style extension of CycleGAN, and train a generator to transfer negative control from one batch to the style of another batch. To further disentangle the biological features (e.g., cell size/density) from the image features (e.g., illumination), we introduce additional regression tasks to make sure the biological evidence is intact during the transformation.…”

Real‐Time Facial Expression Transformation for Monocular RGB Video

“…Our approach builds on the CycleGAN framework [ZPIE17] that learns a mapping function between two unpaired image domains using two GAN models [GPAM*14]. The works of [LZZ16, SLH*17, CCK*17] developed variants of GAN models for face attributes synthesis, such as gender/age modification and expression transformation. However, those imaged‐based GAN models are usually limited to generating low‐resolution images and likely to incur artefacts on face or background change.…”

Unsupervised content-preserving transformation for optical microscopy

“…CycleGAN uses a cycle-consistency learning technique to translate images from one domain to the other one and exhibits comparable performance to supervised methods. Such learning strategy and its modified versions have been validated in style transfer of natural images [28][29][30] and medical image analysis [31][32][33] . As for the field of optical microscopy, a few forward-looking studies have applied cycleGANs to remove coherent noise in optical diffraction tomography 34 , and realize image segmentation for bright-field microscopy and X-ray computed tomography 35 .…”