Real-Time Facial Segmentation and Performance Capture from RGB Input

Abstract

We introduce the concept of unconstrained real-time 3D facial performance capture through explicit semantic segmentation in the RGB input. To ensure robustness, cutting edge supervised learning approaches rely on large training datasets of face images captured in the wild. While impressive tracking quality has been demonstrated for faces that are largely visible, any occlusion due to hair, accessories, or hand-toface gestures would result in significant visual artifacts and loss of tracking accuracy. The modeling of occlusions has been mostly avoided due to its immense space of appearance variability. To address this curse of high dimensionality, we perform tracking in unconstrained images assuming non-face regions can be fully masked out. Along with recent breakthroughs in deep learning, we demonstrate that pixel-level facial segmentation is possible in real-time by repurposing convolutional neural networks designed originally for general semantic segmentation. We develop an efficient architecture based on a two-stream deconvolution network with complementary characteristics, and introduce carefully designed training samples and data augmentation strategies for improved segmentation accuracy and robustness. We adopt a state-of-the-art regressionbased facial tracking framework with segmented face images as training, and demonstrate accurate and uninterrupted facial performance capture in the presence of extreme occlusion and even side views. Furthermore, the resulting segmentation can be directly used to composite partial 3D face models on the input images and enable seamless facial manipulation tasks, such as virtual make-up or face replacement.

State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications

“…Image taken from [HMYL15]. Figure 8: An RGB face tracker that is robust to occlusion has been proposed by [SLL16]. Using deep learning a clean segmentation of all visible parts of the face is obtained.…”

“…Using deep learning a clean segmentation of all visible parts of the face is obtained. Image taken from [SLL16].…”

Robust facial landmark detection and tracking across poses and expressions for in-the-wild monocular video

“…In terms of 3D facial geometry reconstruction for the refinement of landmarks, recently there has been an increasing amount of research based on 2D images and videos [19,[35][36][37][38][39][40][41]. In order to accurately track facial landmarks, it is important to first reconstruct face geometry.…”

“…Robust methods such as Refs. [35,36,39] can track facial performance in the presence of noise but often miss subtle details such as small eyelid and mouth movements, which are important in conveying the target's emotion and to generate convincing animation. Although we use a 3D optical flow approach similar to that in Ref.…”

Realtime Dynamic 3D Facial Reconstruction for Monocular Video In-the-Wild

“…An adaptive scheme was proposed to capture more detail with point-to-point deformation on top of blendshapes in [22]. To explicitly deal with outliers caused by occlusions, a method was proposed to segment the face and complete the occluded parts based on the blendshape in [19], which was later extended to RGB input in [28]. Binocular stereo system, on the other hand, can provide higher resolution and work in outdoor environments directly under sunlight, but are more prone to suffer from lighting variation.…”