Semantic Alignment: Finding Semantically Consistent Ground-Truth for Facial Landmark Detection

Abstract

Recently, deep learning based facial landmark detection has achieved great success. Despite this, we notice that the semantic ambiguity greatly degrades the detection performance. Specifically, the semantic ambiguity means that some landmarks (e.g. those evenly distributed along the face contour) do not have clear and accurate definition, causing inconsistent annotations by annotators. Accordingly, these inconsistent annotations, which are usually provided by public databases, commonly work as the groundtruth to supervise network training, leading to the degraded accuracy. To our knowledge, little research has investigated this problem. In this paper, we propose a novel probabilistic model which introduces a latent variable, i.e. the 'real' ground-truth which is semantically consistent, to optimize. This framework couples two parts (1) training landmark detection CNN and (2) searching the 'real' groundtruth. These two parts are alternatively optimized: the searched 'real' ground-truth supervises the CNN training; and the trained CNN assists the searching of 'real' groundtruth. In addition, to recover the unconfidently predicted landmarks due to occlusion and low quality, we propose a global heatmap correction unit (GHCU) to correct outliers by considering the global face shape as a constraint. Extensive experiments on both image-based (300W and AFLW) and video-based (300-VW) databases demonstrate that our method effectively improves the landmark detection accuracy and achieves the state of the art performance.

Attentive One-Dimensional Heatmap Regression for Facial Landmark Detection and Tracking

“…This is because with the increase of , the quantization error is reduced and more detailed spatial and temporal patterns are captured. For example, as increases from 64 to 768, the NRMSE of our method decreases by 15 com, cha, and full. The light-weight 1D heatmap allows us to fully boost its resolution despite limited space, and the quantization error is well alleviated.…”

“…Chen et al [5] proposed a Conditional Random Field (CRF) method to embed geometric relationships among landmarks based on their heatmaps. Liu et al [15] proposed a heatmap correction unit which uses global shape constraints to refine heatmaps.…”

“…From these approaches, coordinate regression methods include SDM [35], TSCN [26], IFA [1], CFSS [40], TCDCN [38], TSTN [14], DSRN [17], ODN [39], STA [30], Sun et al's work [27] and GAN [36]. Heatmap regression methods include Newell et al's work [18], SAN [9], LAB [32], CNN-CRF [5], LaplaceKL [23], Sun et al's work [28], DSNT [19] , DARK [37], FHR [30], GHCU [15] [10,11,21] for landmark detection are trained under different conditions with our method so are not included in our comparison. Table 3 lists the NRSME performance of the proposed detector and the compared methods on the image datasets, i.e., the 300W public testing set and the AFLW dataset.…”

Robust facial landmark detection by cross-order cross-semantic deep network

“…Implementation Details In our experiments, we first use the four stacked hourglass networks as our backbone [37]. All the training and testing images are In the next section, we firstly compare our algorithm with the state-of-theart methods, such as 3DDFA [44], RAR [29], Wing [36], LAB [8], DU-Net [48], Liu et al [11], ODN [9], HRNet [49], Chandran et al [13] and LUVLi [12].…”

“…For instance, the coordinate regression facial landmark detection methods [6,8,9] learn features from the whole face images and then regress to the landmark coordinates, which drives the models to learn the whole facial features in a common/normal way that cannot accurately model the differences of local details and the relationships among local details. Also, the heatmap regression facial landmark detection methods [10,11,12,13] generate a landmark heatmap for each landmark and then predict landmarks by traversing the corresponding landmark heatmaps.…”