Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation

Abstract

In this work, we propose a novel Cycle In Cycle Generative Adversarial Network (C 2 GAN) for the task of keypoint-guided image generation. The proposed C 2 GAN is a cross-modal framework exploring a joint exploitation of the keypoint and the image data in an interactive manner. C 2 GAN contains two different types of generators, i.e., keypoint-oriented generator and image-oriented generator. Both of them are mutually connected in an end-to-end learnable fashion and explicitly form three cycled sub-networks, i.e., one image generation cycle and two keypoint generation cycles. Each cycle not only aims at reconstructing the input domain, and also produces useful output involving in the generation of another cycle. By so doing, the cycles constrain each other implicitly, which provides complementary information from the two different modalities and brings extra supervision across cycles, thus facilitating more robust optimization of the whole network. Extensive experimental results on two publicly available datasets, i.e., Radboud Faces [19] and Market-1501 [58], demonstrate that our approach is effective to generate more photo-realistic images compared with state-of-the-art models.

A State-of-the-Art Review on Image Synthesis With Generative Adversarial Networks

“…Tang et al [67] proposed a keypoint-guided image generation method called Cycle In Cycle Generative Adversarial Network, which can generate photo-realistic person pose images. Ma et al [68] proposed a person image generation method called Pose Guided Person Generation Network, and it can synthesize high-quality person images with arbitrary poses based on a person image and a pose.…”

Synthesis and Reconstruction of Fingerprints using Generative Adversarial Networks

“…The reconstruction model reconstructs the fingerprint image x given S x . We cast the fingerprint reconstruction as image-to-image translation [29], [30] where the minutiae set is first converted to a minutiae map M x ∈ R h×w×3 following [27], [31], [32]. M x is encoded by a latent vector w∈ R 512 using a Minutiae-To-Style CNN (Section III-C), that is fed to the pretrained generator G to reconstruct the input image x.…”

RFGAN: RF-Based Human Synthesis

“…Therefore, we modify our model with some classic approaches that are widely used in GANs or related works, and the modified models are set as the baselines: -Img&RF: To enable the RF-based condition setting, we propose a RF-Extractor with RNN to encode RF heatmaps and use RF-InNorm to inject the extracted information. Another alternative approach is to concatenate the RF condition with the input image directly, which is effective when the conditions have explicit guidance for GAN, e.g., pose-guided human synthesis [25]. However, the RF conditions are obscure data and have totally different spatial structures with optical images.…”

“…Researchers have explored various kinds of conditions, e.g., category labels [19], text descriptions [20]- [22], and images [23]- [31]. From technology perspective, most existing GAN models require the conditions to be able to guide the GAN model explicitly [19], [23], [25], [26], [31] or can be transformed to conditional variables for GAN using an existing pre-trained model [20]- [22].…”