DA-GAN: Instance-Level Image Translation by Deep Attention Generative Adversarial Networks

Abstract

Unsupervised image translation, which aims in translating two independent sets of images, is challenging in discovering the correct correspondences without paired data. Existing works build upon Generative Adversarial Network (GAN) such that the distribution of the translated images are indistinguishable from the distribution of the target set. However, such set-level constraints cannot learn the instance-level correspondences (e.g. aligned semantic parts in object configuration task). This limitation often results in false positives (e.g. geometric or semantic artifacts), and further leads to mode collapse problem. To address the above issues, we propose a novel framework for instance-level image translation by Deep Attention GAN (DA-GAN). Such a design enables DA-GAN to decompose the task of translating samples from two sets into translating instances in a highly-structured latent space. Specifically, we jointly learn a deep attention encoder, and the instancelevel correspondences could be consequently discovered through attending on the learned instance pairs. Therefore, the constraints could be exploited on both set-level and instance-level. Comparisons against several state-ofthe-arts demonstrate the superiority of our approach, and the broad application capability, e.g, pose morphing, data augmentation, etc., pushes the margin of domain translation problem.

Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss

“…Attention mechanism is an emerging topic in natural language tasks [20] and image/video generation task [26,37,22,36]. Pumarola et al [26] generated facial expression conditioned on action units annotations.…”

Object-Driven Text-To-Image Synthesis via Adversarial Training

“…Although images with realistic texture have been synthesized on simple datasets, such as birds [29,16] and flowers [33], most existing approaches do not specifically model objects and their relations in images and thus have difficulties in generating complex scenes such as those in the COCO dataset [15]. For example, generating images from a sentence "several people in their ski gear are in the snow" requires modeling of different objects (people, ski gear) and their interactions (people on top of ski gear), as well as filling the missing information (e.g., the rocks in the background).…”

Multi-Channel Attention Selection GAN With Cascaded Semantic Guidance for Cross-View Image Translation

“…To further improve the generation performance, the attention mechanism has been recently investigated in image translation, such as [3,45,39,24,26]. However, to the best of our knowledge, our model is the first attempt to incorporate a multi-channel attention selection module within a GAN framework for image-to-image translation task.…”