Occlusion-Aware 3D Morphable Models and an Illumination Prior for Face Image Analysis

Abstract

Faces in natural images are often occluded by a variety of objects. We propose a fully automated, probabilistic and occlusion-aware 3D Morphable Face Model adaptation framework following an Analysis-by-Synthesis setup. The key idea is to segment the image into regions explained by separate models. Our framework includes a 3D Morphable Face Model, a prototypebased beard model and a simple model for occlusions and background regions. The segmentation and all the model parameters have to be inferred from the single target image. Face model adaptation and segmentation are solved jointly using an expectation-maximizationlike procedure. During the E-step, we update the segmentation and in the M-step the face model parameters are updated. For face model adaptation we apply a stochastic sampling strategy based on the Metropolis-Hastings algorithm. For segmentation, we apply Loopy Belief Propagation for inference in a Markov random field. Illumination estimation is critical for occlusion handling. Our combined segmentation and model adaptation needs a proper initialization of the illumination parameters. We propose a RANSAC-based robust illumination estimation technique. By applying this method to a large face image database we obtain a first empirical distribution of real-world illumination conditions. The obtained empirical distribution is made publicly available and can be used as prior in probabilistic frameworks, for regularization or to synthesize data for deep learning methods.

Visibility Constrained Generative Model for Depth-Based 3D Facial Pose Tracking

“…A different modeling strategy involves rigid and nonrigid registration of 3D face models to the depth images, either through the use of 3D morphable models [2], [11], [12], [25], [34], [35], [36], [37], [38], [39], [40], [41], or brute-force per-vertex 3D face reconstruction [14], [14], [15], [16], [25], [42]. Although such systems may be accurate, they often require offline initialization or user calibration to create face models specific to individual users.…”

“…Our proposed method falls into this category. Inspired by statistical models [2], [25], [26], [41], we enhance this face model through a decomposable statistical formulation, in which the shape variations from identity and expression are explicitly disentangled.…”

“…R OBUST 3D facial pose tracking is a central task in many computer vision and computer graphics problems, with applications in facial performance capture, humancomputer interaction, as well as VR/AR applications in modern mobile devices. Although the facial pose tracking has been successfully performed on RGB data [1], [2], [3], [4], [5], [6], [7], [8], [9], [10] for well-constrained scenes, challenges posed by illumination variations, shadows, and substantial occlusions make these RGB-based facial pose tracking approaches less reliable in unconstrained scenarios. The utilization of depth data from commodity real-time range sensors has led to more robust 3D facial pose tracking, not only by enabling registration in the 3D metric space but also by providing cues for the occlusion reasoning.…”

“…Several key challenges need to be addressed in the context of depth-based tracking: (1) coping with complex self-occlusions and other occlusions caused by hair, accessories, hands and etc. ; (2) sustaining an always-on face tracker that can dynamically adapt to any user without manual recalibration; and (3) providing stability over time to variations in user expressions. Unlike previous depthbased discriminative or data-driven methods [18], [19], [20], [21], [22], [23], [24] that require complex training or manual calibration, in this paper we propose a framework that unifies pose tracking and face model adaptation on-the-fly, offering highly accurate, occlusion-aware and uninterrupted 3D facial pose tracking, as shown in Fig.…”

“…arXiv:1905.02114v1 [cs.CV] 6 May 2019 2) We propose an occlusion-aware pose estimation mechanism based on minimizing an information-theoretic ray visibility score that optimizes the visibility of our statistical face model. The mechanism is based on the intuition that valid poses imply that the face points must either be colocated (i.e., visible) with the observed point cloud or be occluded by the point cloud.…”