3D Face Morphable Models "In-the-Wild"

Abstract

Abstract3D Morphable Models (3DMMs) are powerful statistical models of 3D facial shape and texture, and among the stateof-the-art methods for reconstructing facial shape from single images. With the advent of new 3D sensors, many 3D facial datasets have been collected containing both neutral as well as expressive faces. However, all datasets are captured under controlled conditions. Thus, even though powerful 3D facial shape models can be learnt from such data, it is difficult to build statistical texture models that are sufficient to reconstruct faces captured in unconstrained conditions ("in-the-wild"). In this paper, we propose the first, to the best of our knowledge, "in-the-wild" 3DMM by combining a powerful statistical model of facial shape, which describes both identity and expression, with an "in-the-wild" texture model. We show that the employment of such an "in-thewild" texture model greatly simplifies the fitting procedure, because there is no need to optimize with regards to the illumination parameters. Furthermore, we propose a new fast algorithm for fitting the 3DMM in arbitrary images. Finally, we have captured the first 3D facial database with relatively unconstrained conditions and report quantitative evaluations with state-of-the-art performance. Complementary qualitative reconstruction results are demonstrated on standard "in-the-wild" facial databases. An open source implementation of our technique is released as part of the Menpo Project [1].

The 3D Menpo Facial Landmark Tracking Challenge

“…• We provide a large scale database of facial images with 3DA-2D and 3D facial landmarks by applying the state-of-the-art 3DMM fitting algorithm of [1] driven by the ground-truth 2D landmarks.…”

Recovering Joint and Individual Components in Facial Data

“…More specifically, the shape model describes facial meshes that consist of L vertexes and is built by applying dense registration on a set of training meshes followed by PCA [29]. An instance of the shape model can be expressed as the linear combination of a mean shapes and the subspace U s with parameters p as s =s+U s p. Similarly, the texture model is a linear PCA model that describes the texture associated with the shape model and can be constructed from captured 3D texture as in [29], or from single 2D images as in [30]. Moreover, the camera model maps a 3D mesh on the image plane, utilizing an orthographic or a perspective transformation W (p, c), where c are the camera parameters.…”

“…Fitting a 3DMM into a new image is an iterative process, where the model parameters (regarding shape, texture, and camera) are updated at each iteration. Typically, the fitting procedure is formulated as a Gauss-Newton optimization problem, where the main task is the minimization of the error between the input and the reconstructed image [30]. The extraction of 3D texture from single images commences with fitting a 3DMM on them.…”

“…Similarly to the 2D face age progression experiments presented previously, the AgeDB database was divided into M = 6 age groups ('21-30', '31-40', '41-50', '51-60', '61-70', '71+') and used to train the RJIVE-M. In order to acquire the 3D training data for this task the 3DMM-ITW [30] was employed. The optimal shape and camera parameters were extracted by fitting the model to each one of the images of all age groups.…”

“…In the test phase, the 3D shape of the test face is obtained by using the 3DMM-ITW algorithm [30]. Then the UV texture and the corresponding self-occlusion mask are computed by employing the recovered 3D shape.…”