Dynamic 3D avatar creation from hand-held video input

Abstract

EPFLFigure 1: Our system creates a fully rigged 3D avatar of the user from uncalibrated video input acquired with a cell-phone camera. The blendshape models of the reconstructed avatars are augmented with textures and dynamic detail maps, and can be animated in realtime. AbstractWe present a complete pipeline for creating fully rigged, personalized 3D facial avatars from hand-held video. Our system faithfully recovers facial expression dynamics of the user by adapting a blendshape template to an image sequence of recorded expressions using an optimization that integrates feature tracking, optical flow, and shape from shading. Fine-scale details such as wrinkles are captured separately in normal maps and ambient occlusion maps. From this user-and expression-specific data, we learn a regressor for on-the-fly detail synthesis during animation to enhance the perceptual realism of the avatars. Our system demonstrates that the use of appropriate reconstruction priors yields compelling face rigs even with a minimalistic acquisition system and limited user assistance. This facilitates a range of new applications in computer animation and consumer-level online communication based on personalized avatars. We present realtime application demos to validate our method.

3D Character Animation: A Brief Review

“…For blend shape animation, linearly blending of template shapes would fail to realize fine-scale details such as winkle. Recent studies support this hypothesis using Detail-maps for shading (34). For skinning, applying non-rigid deformation to a character is hard.…”

Robust facial landmark detection and tracking across poses and expressions for in-the-wild monocular video

“…In terms of 3D facial geometry reconstruction for the refinement of landmarks, recently there has been an increasing amount of research based on 2D images and videos [19,[35][36][37][38][39][40][41]. In order to accurately track facial landmarks, it is important to first reconstruct face geometry.…”

“…In order to accurately track facial landmarks, it is important to first reconstruct face geometry. Due to the lack of depth information in images and videos, most methods rely on blendshape priors to model nonrigid deformation while structure-from-motion, photometric stereo, or other methods [42] are used to account for unseen variation [36,38] or details [19,37].…”

Realtime Dynamic 3D Facial Reconstruction for Monocular Video In-the-Wild

“…The creation of photo-realistic person specific facial model can be useful for many application as seen in [34,23], where the tracked facial performance was used to transfer the expression of source actor to the target. In order to create person specific model from a monocular rig, a method was proposed in [20], which produces facial mesh via multi-view stereo vision pipeline. To allow the geometry deformation and variations go beyond the blendshape and minuscule details, a method was proposed in [38], which physically models the anatomical structure of the face and deform the person specific model to match the monocular video input.…”

“…2b). SFM methods [18,11,20], which estimate 3D structures from 2D images with different viewing angles, are able to handle these large variations. Nevertheless, the user is required to remain still while images from different angles are being taken.…”