Real-time high-fidelity facial performance capture

Abstract

We present the first real-time high-fidelity facial capture method. The core idea is to enhance a global real-time face tracker, which provides a low-resolution face mesh, with local regressors that add in medium-scale details, such as expression wrinkles. Our main observation is that although wrinkles appear in different scales and at different locations on the face, they are locally very self-similar and their visual appearance is a direct consequence of their local shape. We therefore train local regressors from high-resolution capture data in order to predict the local geometry from local appearance at runtime. We propose an automatic way to detect and align the local patches required to train the regressors and run them efficiently in real-time. Our formulation is particularly designed to enhance the low-resolution global tracker with exactly the missing expression frequencies, avoiding superimposing spatial frequencies in the result. Our system is generic and can be applied to any real-time tracker that uses a global prior, e.g. blend-shapes. Once trained, our online capture approach can be applied to any new user without additional training, resulting in high-fidelity facial performance reconstruction with person-specific wrinkle details from a monocular video camera in real-time.

Extending implicit skinning with wrinkles

“…Therefore the curves are not grown again from their associated seed after they are generated. This is in line with the observation that for human skin in particular, wrinkles tend to form at the same position for multiple repetitions of the same movement (fingers are a typical example), a feature exploited for instance by [4] for their wrinkle likelihood map generation.…”

“…It would also be useful to investigate ways to make the convolution surfaces look less cylindrical and more organic, not restriced only to circle profiles; for example noise could be added, or segments with different radius could be layered. Finally, another interesting future direction inspired by [4] would be to combine the proceduralism of the method with a trained approach to learn the space of input parameters for a closer match to physical appearance of acquired data. …”

A deep learning approach for generalized speech animation

“…The advantage of hand animation is that the artist can precisely style and time the animation, but it is extremely costly and time consuming to produce. The main alternative to hand animation is performance-driven animation using facial motion capture of an actor's face [Beeler et al 2011;Cao et al 2015Cao et al , 2013Fyfe et al 2014;Huang et al 2011;Li et al 2013;Weise et al 2011;Weng et al 2014;Zhang et al 2004]. Performance-driven animation requires an actor to perform all shots, and may generate animation parameters that are complex and time consuming for an animator to edit (e.g.…”

Realtime Dynamic 3D Facial Reconstruction for Monocular Video In-the-Wild

“…Later a dynamic method was proposed to automatically track and generate personalized facial blendshape in realtime in [4]. Built on top of the robust tracking, more details were added to the person specific blendshape based on image input and user interaction in [3,7]. The creation of photo-realistic person specific facial model can be useful for many application as seen in [34,23], where the tracked facial performance was used to transfer the expression of source actor to the target.…”

“…Existing works based on parametric 3D facial model and shape-from-shading [32,3,14] are able to reconstruct minuscule detail while allowing the user to move around freely in the monocular setting. However, these methods cannot deal with structures such as hair and glasses (Fig.…”