Free-Viewpoint Facial Re-Enactment from a Casual Capture

Rao, Srinivas; Ortiz-Cayon, Rodrigo; Munaro, Matteo; Liaudanskas, Aidas; Chande, Krunal; Bertel, Tobias; Richardt, Christian; Alexander, José; Hölzer, Stefan M.; Kar, Abhishek

doi:10.1145/3415264.3425453

Cited by 5 publications

(3 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our approach allows true freeviewpoint navigation while reasonably preserving identiy. Rao et al [2020] shares some similarities with our approach, since they also process facial animations in a canonical frame, and lift the face to 3D. However, they use StyleGAN to synthesize only the mouth, and require dense capture.…”

Section: Face Models and Portrait Renderingmentioning

confidence: 99%

See 1 more Smart Citation

FreeStyleGAN

Leimkühler

Drettakis

2021

ACM Trans. Graph.

View full text Add to dashboard Cite

Current Generative Adversarial Networks (GANs) produce photorealistic renderings of portrait images. Embedding real images into the latent space of such models enables high-level image editing. While recent methods provide considerable semantic control over the (re-)generated images, they can only generate a limited set of viewpoints and cannot explicitly control the camera. Such 3D camera control is required for 3D virtual and mixed reality applications. In our solution, we use a few images of a face to perform 3D reconstruction, and we introduce the notion of the GAN camera manifold, the key element allowing us to precisely define the range of images that the GAN can reproduce in a stable manner. We train a small face-specific neural implicit representation network to map a captured face to this manifold and complement it with a warping scheme to obtain free-viewpoint novel-view synthesis. We show how our approach - due to its precise camera control - enables the integration of a pre-trained StyleGAN into standard 3D rendering pipelines, allowing e.g., stereo rendering or consistent insertion of faces in synthetic 3D environments. Our solution proposes the first truly free-viewpoint rendering of realistic faces at interactive rates, using only a small number of casual photos as input, while simultaneously allowing semantic editing capabilities, such as facial expression or lighting changes.

show abstract

Section: Face Models and Portrait Renderingmentioning

confidence: 99%

“…Since our camera manifold assumes the eyes to be at defined 3D positions p 𝑙 and p 𝑟 , we perform 3D alignment to the canonical configuration (Eq. 1) [Gao et al 2020;Rao et al 2020], making our method independent of the reconstruction algorithm.…”

Section: Alignment To Canonical Coordinatesmentioning

confidence: 99%