Disentangling Geometry and Appearance with Regularised Geometry-Aware Generative Adversarial Networks

Abstract

Deep generative models have significantly advanced image generation, enabling generation of visually pleasing images with realistic texture. Apart from the texture, it is the shape geometry of objects that strongly dictates their appearance. However, currently available generative models do not incorporate geometric information into the image generation process. This often yields visual objects of degenerated quality. In this work, we propose a regularized Geometry-Aware Generative Adversarial Network (GAGAN) which disentangles appearance and shape in the latent space. This regularized GAGAN enables the generation of images with both realistic texture and shape. Specifically, we condition the generator on a statistical shape prior. The prior is enforced through mapping the generated images onto a canonical coordinate frame using a differentiable geometric transformation. In addition to incorporating geometric information, this constrains the search space and increases the model's robustness. We show that our approach is versatile, able to generalise across domains (faces, sketches, hands and cats) and sample sizes (from as little as ∼ 200−30,000 to more than 200,000). We demonstrate superior performance through extensive quantitative and qualitative experiments in a variety of tasks and settings. Finally, we leverage our model to automatically and accurately detect errors or drifting in facial landmarks detection and tracking in-the-wild.

Unsupervised Controllable Generation with Self-Training

“…[26]. While the majority of existing work on disentanglement focuses on a (semi-)supervised setting [27][28][29][30][31][32][33][34], our work focuses on the unsupervised seeting. Here, we review the most closely related methods below.…”

Policy-Gradient and Actor-Critic Based State Representation Learning for Safe Driving of Autonomous Vehicles

“…Deep learning techniques have gained increased attention as their design requires minimal prior knowledge and the models can be fine-tuned to scale to different environments [ 20 ]. These models have been enhanced using recurrent neural networks (RNN) that memorize long-term dependencies and tackle autonomous driving as partially observable Markov decision processes (POMDP) [ 21 ].…”

“…All past and present states are in the continuous state-space [ 34 ]. The parameter values, that is, the set of velocities in a given timestep, at present and past instances for a given state S are obtained for the trajectory followed by the vehicle described by [ 20 , 40 ]: where is the difference between two subsequent timeframes while the vehicle navigates the trajectory. These parameters are used to calculate the optimal value function and optimal Q-value .…”

“…All past and present states are in the continuous state-space [34]. The parameter values, that is, the set of velocities in a given timestep, at present and past instances for a given state S are obtained for the trajectory followed by the vehicle described by [20,40]:…”

RoCGAN: Robust Conditional GAN

“…Conditional signal generation leverages a conditioning label, e.g. a prior shape (Tran et al 2019) or an embedded representation (Mirza and Osindero 2014), to produce the target signal. In this work, we focus on the latter setting, i.e.…”