Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

Abstract

Differentiable rendering has paved the way to training neural networks to perform "inverse graphics" tasks such as predicting 3D geometry from monocular photographs. To train high performing models, most of the current approaches rely on multi-view imagery which are not readily available in practice. Recent Generative Adversarial Networks (GANs) that synthesize images, in contrast, seem to acquire 3D knowledge implicitly during training: object viewpoints can be manipulated by simply manipulating the latent codes. However, these latent codes often lack further physical interpretation and thus GANs cannot easily be inverted to perform explicit 3D reasoning. In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers. Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN's latent code into interpretable 3D properties. The entire architecture is trained iteratively using cycle consistency losses. We show that our approach significantly outperforms state-of-the-art inverse graphics networks trained on existing datasets, both quantitatively and via user studies. We further showcase the disentangled GAN as a controllable 3D "neural renderer", complementing traditional graphics renderers. * indicates equal contribution.

Lifting 2D StyleGAN for 3D-Aware Face Generation

“…However, given the large degrees of freedom of such representation and the noisy signals from adversarial training, the output shapes of their work suffer from strong distortion. Concurrent with our work, Zhang et al [33] and Pan et al [21] have utilized StyleGAN to generate multiview synthetic data for 3D reconstruction tasks. Zhang et al [33] conduct manual annotation on offline-generated data while Pan et al [21] propose to iteratively synthesize data and train the reconstruction network.…”

“…Concurrent with our work, Zhang et al [33] and Pan et al [21] have utilized StyleGAN to generate multiview synthetic data for 3D reconstruction tasks. Zhang et al [33] conduct manual annotation on offline-generated data while Pan et al [21] propose to iteratively synthesize data and train the reconstruction network. Different from their work, our work builds a 3D generative model by simultaneously learning to manipulate StyleGAN2 generation and estimate 3D shapes.…”

“…Here, since we already have a pre-trained generator, we utilize the G 2D as a multiview data generator to provide supervision for rotated-views of the 3D face. This approach is inspired by recent findings that StyleGAN2 is able to generate different views of a target sample by changing its style code [24,25,33]. Formally, for each rendered sample I w = R(A, S, T, V, L), we randomly sample a different view point V and lighting L to render a rotated and relighted face image I w = R(A, S, T, V , L ).…”

A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond

“…Besides, generative adversarial networks have proven successful at generating photo-realistic images in many domains [63]. Recent work on neural rendering and inverse graphics [142] suggests that those generative models learn representations in which geometry, light and texture can be disentangled. Such approaches could be used to generate new synthetic data samples that allow for larger intrinsic decomposition datasets.…”

EditGAN: High-Precision Semantic Image Editing

“…GAN's latent space or works directly with GAN-generated images. Careful modifications of the latent embeddings then translate to desired changes in generated output, allowing, for example, to coherently change facial expressions in portraits [9][10][11][12][13][14][15][16], change viewpoint or shapes and textures of cars [17], or to interpolate between different images in a semantically meaningful manner [18][19][20][21].…”