FML: Face Model Learning From Videos

Abstract

Figure 1. We propose multi-frame self-supervised training of a deep network based on in-the-wild video data for jointly learning a face model and 3D face reconstruction. Our approach successfully disentangles facial shape, appearance, expression, and scene illumination. AbstractMonocular image-based 3D reconstruction of faces is a long-standing problem in computer vision. Since image data is a 2D projection of a 3D face, the resulting depth ambiguity makes the problem ill-posed. Most existing methods rely on data-driven priors that are built from limited 3D face scans. In contrast, we propose multi-frame video-based self-supervised training of a deep network that (i) learns a face identity model both in shape and appearance while (ii) jointly learning to reconstruct 3D faces. Our face model is learned using only corpora of in-the-wild video clips collected from the Internet. This virtually endless source of training data enables learning of a highly general 3D face model. In order to achieve this, we propose a novel multiframe consistency loss that ensures consistent shape and appearance across multiple frames of a subject's face, thus minimizing depth ambiguity. At test time we can use an arbitrary number of frames, so that we can perform both monocular as well as multi-frame reconstruction.

Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion

“…. Several other works have shown that combining a prior template about the object category shape with video allows for an improved 3D reconstruction of the underlying geometry, both for faces [67,63,43] and quadrupeds [8]. However, these methods still require multiple videos and a template, while our method does not.…”

“…Effectively all works addressing aspects related to 3D geometry rely on paired data for training, e.g. multiple views of the same object [71], videos [48] or some pre-existing 3D mesh representation that is the starting point for further disentanglement [21,56,81,62] or self-supervision [85].…”

3D Morphable Face Models—Past, Present, and Future

“…Recently, several deep learning-based models were published that fall into this group of nonlinear models [Bagautdinov et al 2018;Lombardi et al 2018;Tewari et al 2019Tran and Liu 2018a]. Section 6 covers these models in more detail.…”

“…The rise of deep learning methods facilitated to learn per-vertex appearance models directly from images, such as done by , who learn per-vertex albedo model offsets in order to improve the generalization ability of an existing PCA-based model. Similarly, Tewari et al [2019], learn a per-vertex albedo model from scratch based on video data. train a mesh decoder that jointly models the texture and shape on a per-vertex basis, which, however, relies on the availability of 3D shape and appearance data.…”

360-Degree Textures of People in Clothing from a Single Image

“…Hence, in this work, we learn from complete texture maps obtained from 3D registrations. 3D person reconstruction from images While promising, recent methods for 3D person reconstruction either require video as input [6,7,8], scans [74], do not allow control over pose, shape and clothing [48,56], focus only on faces [72,32,63,57,47,62], or only on garments [68].…”