Model-Assisted 3D Face Reconstruction from Video

Abstract

Abstract. This paper describes a model-assisted system for reconstruction of 3D faces from a single consumer quality camera using a structure from motion approach. Typical multi-view stereo approaches use the motion of a sparse set of features to compute camera pose followed by a dense matching step to compute the final object structure. Accurate pose estimation depends upon precise identification and matching of feature points between images, but due to lack of texture on large areas of the face, matching is prone to errors.To deal with outliers in both the sparse and dense matching stages, previous work either relies on a strong prior model for face geometry or imposes restrictions on the camera motion. Strong prior models result in a serious compromise in final reconstruction quality and typically bear a signature resemblance to a generic or mean face. Model-based techniques, while giving the appearance of face detail, in fact carry this detail over from the model prior. Face features such as beards, moles, and other characteristic geometry are lost. Motion restrictions such as allowing only pure rotation are nearly impossible to satisfy by the end user, especially with a handheld camera.We significantly improve the robustness and flexibility of existing monocular face reconstruction techniques by introducing a deformable generic face model only at the pose estimation, face segmentation, and preprocessing stages. To preserve data fidelity in the final reconstruction, this generic model is discarded completely and dense matching outliers are removed using tensor voting: a purely data-driven technique. Results are shown from a complete end to end system.

A SfM-based 3D face reconstruction method robust to self-occlusion by using a shape conversion matrix

“…If the optimal model parameters are found, a detailed 3D facial shape can then be reconstructed from these model parameters. However, these methods require a high computational costs and possibly fall into a local minima because the number of model parameters is considerably large [8,9]. If they fall into a local minima, the reconstructed 3D face tends to be close to the mean face because the parameter optimization begins at the mean face [8,9].…”

“…However, these methods require a high computational costs and possibly fall into a local minima because the number of model parameters is considerably large [8,9]. If they fall into a local minima, the reconstructed 3D face tends to be close to the mean face because the parameter optimization begins at the mean face [8,9]. In order to reduce the computational complexities of the original 3D morphable model, simplified morphable models representing only facial shape have been proposed in [3,[10][11][12].…”

“…Compared to model-based methods, these methods do not require the parameter optimization started from the mean of the 3D faces. As a result, a personspecific 3D facial shape can be reconstructed by using SfM [8,9].…”

“…SfM-based methods estimate a 3D facial shape and a projection matrix from the corresponding 2D facial feature points (FFPs) of multiple facial images [8,9,[13][14][15][16]. The basic idea of SfM is that the corresponding 2D FFPs can be factorized into the 3D facial shape and the projection matrix by using some reasonable constraints, such as the rank constraint [17].…”

Accurate 3D face reconstruction from weakly calibrated wide baseline images with profile contours

“…For dense reconstruction, we employ a face-specific cylindrical representation, which allows us to solve a global optimization problem for N-view dense aggregation. In [12], individual pairwise reconstructions are integrated into a single 3D point cloud, and then an additional postprocessing (e.g. tensor voting) is applied to enforce surface self-consistency.…”