Guiding GANs: How to control non-conditional pre-trained GANs for conditional image generation

Abstract

Generative Adversarial Networks (GANs) are an arrange of two neural networks -the generator and the discriminator-that are jointly trained to generate artificial data, such as images, from random inputs. The quality of these generated images has recently reached such levels that can often lead both machines and humans into mistaking fake for real examples. However, the process performed by the generator of the GAN has some limitations when we want to condition the network to generate images from subcategories of a specific class. Some recent approaches tackle this conditional generation by introducing extra information prior to the training process, such as image semantic segmentation or textual descriptions. While successful, these techniques still require defining beforehand the desired subcategories and collecting large labeled image datasets representing them to train the GAN from scratch. In this paper we present a novel and alternative method for guiding generic non-conditional GANs to behave as conditional GANs. Instead of re-training the GAN, our approach adds into the mix an encoder network to generate the high-dimensional random input vectors that are fed to the generator network of a non-conditional GAN to make it generate images from a specific subcategory. In our experiments, when compared to training a conditional GAN from scratch, our guided GAN is able to generate artificial images of perceived quality comparable to that of non-conditional GANs after training the encoder on just a few hundreds of images, which substantially accelerates the process and enables adding new subcategories seamlessly.

Flow Plugin Network for conditional generation

“…Some of them extend traditional VAE architectures in order to construct the conditional variants of this kind of model for tasks like visual segmentation [3], conditional image [12] or text generation [13]. Conditional variants of GANs (cGANs) [4] are also popular for class-specific image [14] or point cloud [15] generation. Conditioning mechanisms are also implemented in some variants of flow models including Masked Autoregressive Flows (MAF) [10], RealNVP [11] and continuous normalizing flows [16].…”

“…To perform such a conditional generation we need to put an additional effort to create a new model with such a functionality. In the case of images, specific solutions were proposed: Conditional Image Generation, with conditional modification of the well-known unconditional generative models -Conditional Variational Autoencoders [3] and Conditional Generative Adversarial Networks [4]. These approaches provide a good result in conditional image generation.…”