Head Pose Estimation through Keypoints Matching between Reconstructed 3D Face Model and 2D Image

Abstract

Mainstream methods treat head pose estimation as a supervised classification/regression problem, whose performance heavily depends on the accuracy of ground-truth labels of training data. However, it is rather difficult to obtain accurate head pose labels in practice, due to the lack of effective equipment and reasonable approaches for head pose labeling. In this paper, we propose a method which does not need to be trained with head pose labels, but matches the keypoints between a reconstructed 3D face model and the 2D input image, for head pose estimation. The proposed head pose estimation method consists of two components: the 3D face reconstruction and the 3D–2D matching keypoints. At the 3D face reconstruction phase, a personalized 3D face model is reconstructed from the input head image using convolutional neural networks, which are jointly optimized by an asymmetric Euclidean loss and a keypoint loss. At the 3D–2D keypoints matching phase, an iterative optimization algorithm is proposed to match the keypoints between the reconstructed 3D face model and the 2D input image efficiently under the constraint of perspective transformation. The proposed method is extensively evaluated on five widely used head pose estimation datasets, including Pointing’04, BIWI, AFLW2000, Multi-PIE, and Pandora. The experimental results demonstrate that the proposed method achieves excellent cross-dataset performance and surpasses most of the existing state-of-the-art approaches, with average MAEs of 4.78∘ on Pointing’04, 6.83∘ on BIWI, 7.05∘ on AFLW2000, 5.47∘ on Multi-PIE, and 5.06∘ on Pandora, although the model of the proposed method is not trained on any of these five datasets.

Deep Learning for Head Pose Estimation: A Survey

“…There is no learning involved for head pose that is estimated from the 3D deformable model by minimizing the projection error for all landmark points. Liu et al [85] trained a CNN to reconstruct a personalized 3D face model from the input head image and through an iterative 3D-2D keypoints matching algorithm estimate head pose under constraint perspective transformation (see Fig. 8).…”

“…The network can be directly Fig. 8 An example of deformable model: A personalized 3D face is reconstructed from the input head image using a CNN, then keypoints matching is used to obtain the pose [85] used for pose estimation, indeed, 3DMM regression contains pose, shape and expression parameters. There is no keypoints matching involved.…”

DOPE++: 6D pose estimation algorithm for weakly textured objects based on deep neural networks

“…In recent years, 6D pose estimation networks based on deep learning have excelled in terms of accuracy and efficiency [ 20 , 29 – 31 ]. However, the lack of real data for training the network makes it difficult to expand the network to new application scenarios, such as in the field of smart manufacturing [ 32 – 34 ] and autonomous driving [ 35 , 36 ].…”

Robust Estimation and Optimized Transmission of 3D Feature Points for Computer Vision on Mobile Communication Network

“…This method finds Harris corner points in various scales and detects robust points for scale variants. Shi and Tomasi proposed the Shi–Tomasi corner considering an affine transformation [ 43 , 44 ]. The most well-known method is Lowe’s SIFT (Scale Invariant Feature Transform) [ 45 ].…”