HeadFusion: 360 Head Pose Tracking Combining 3D Morphable Model and 3D Reconstruction

Abstract

Head pose estimation is a fundamental task for face and social related research. Although 3D morphable model (3DMM) based methods relying on depth information usually achieve accurate results, they usually require frontal or mid-profile poses which preclude a large set of applications where such conditions can not be garanteed, like monitoring natural interactions from fixed sensors placed in the environment. A major reason is that 3DMM models usually only cover the face region. In this paper, we present a framework which combines the strengths of a 3DMM model fitted online with a prior-free reconstruction of a 3D full head model providing support for pose estimation from any viewpoint. In addition, we also proposes a symmetry regularizer for accurate 3DMM fitting under partial observations, and exploit visual tracking to address natural head dynamics with fast accelerations. Extensive experiments show that our method achieves state-of-the-art performance on the public BIWI dataset, as well as accurate and robust results on UbiPose, an annotated dataset of natural interactions that we make public and where adverse poses, occlusions or fast motions regularly occur.

Visibility Constrained Generative Model for Depth-Based 3D Facial Pose Tracking

“…A different modeling strategy involves rigid and nonrigid registration of 3D face models to the depth images, either through the use of 3D morphable models [2], [11], [12], [25], [34], [35], [36], [37], [38], [39], [40], [41], or brute-force per-vertex 3D face reconstruction [14], [14], [15], [16], [25], [42]. Although such systems may be accurate, they often require offline initialization or user calibration to create face models specific to individual users.…”

“…The overall formulation for the rigid pose tracking combines the ray visibility score and the temporal constraint, as L(∆θ) = L rvs (θ (t−1) + ∆θ) + L t (∆θ) + L s (∆θ). (16) We seek to estimate the incremental pose parameters ∆θ between adjacent frames rather than the absolute poses θ between the canonical face model and the input point cloud.…”

“…Although promising results have been demonstrated by leveraging both RGB and depth data [11], [12], [13], [14], [15], [16], [17], or even RGB data alone [2], [6] in unconstrained facial pose tracking, existing approaches are not yet able to reliably cope with RGB data affected by inconsistent or poor lighting conditions. For example, mobile applications like FaceID or Animoji (which heavily employ the face tracking module in their systems) are bound to fail in dark scenes such as bedrooms and theaters, or scenes under complex illuminations such as parties and clubs.…”

Guest Editorial: The Computational Face

“…Another task vastly represented in the computational face special issue is that of head pose estimation [11], [12], [13], [14]. Booth et al introduce a new model for facial shape reconstruction in the wild [11].…”

“…In addition, they release a new data set for facial shape reconstruction in the wild. A similar approach that combines a 3DMM adapted online from samples and a 3D head model similar to the one used for KinectFusion is proposed by Yu et al [12]. The combination of models proves to be helpful for head pose estimation and tracking when people is not directly in front the camera.…”