Track Facial Points in Unconstrained Videos

Abstract

Tracking Facial Points in unconstrained videos is challenging due to the non-rigid deformation that changes over time. In this paper, we propose to exploit incremental learning for person-specific alignment in wild conditions.Our approach takes advantage of part-based representation, as illustrated in Figure 1 and cascade regression for robust and efficient alignment on each frame. Unlike existing methods that usually rely on models trained offline, we incrementally update the representation subspace and the cascade of regressors in a unified framework to achieve personalized modeling.Blind model adaptation without correction would inevitably result in model drifting. How to effectively detect misalignment is still a challenging question that is seldom investigated. To address this issue, we propose a deep neural network for robust fitting evaluation to pick out well-aligned faces from misalignment. The architecture of the network is shown is Figure 2. These well-aligned faces are then used to incrementally update the representation subspace and fitting strategy for person-specific modeling on the fly. In summary, our work makes the following contributions:(1) We propose a novel approach for sequential face alignment. To the best of our knowl- edge, this is the first time that person-specific modeling is investigated to jointly learn the representation subspace and the fitting parameters in a unified framework.(2) The proposed part-based representation together with the cascade regression guarantees robust alignment in unconstrained conditions. More importantly, they are critical to efficiently construct personalized models for real-time or large-scale applications.(3) We propose to leverage deep neural networks for efficient and robust fitting evaluation. It significantly alleviates the drifting issue which would severely deteriorate learned models and inevitably lead to failure.To validate our approach, we provide a detailed experimental analysis of each component of our approach, as well as performance comparisons with existing approaches. Four image datasets (MultiPIE, LFPW, Helen, and AFLW), four video datasets (FGNET, ASLV, 300-VW, and YtbVW), and four state of the arts (RLMS, DRMF, IFA, and ESR) are employed to conduct the experiments. The results demonstrate that the proposed incremental learning can significantly improve the fitting accuracy with an affordable computational cost, especially in unconstrained videos with extensive variations.

A Review of Facial Landmark Extraction in 2D Images and Videos Using Deep Learning

“…Considering all the methods, one of the main ones designed a pose-specific CRM [68], and another one used a progressive initialization [69], which aims to improve the problem of initialization in extremely bad poses. Taking inspiration from the incremental learning method [70], Peng et al [71] used a CNN with likelihood maps to evaluate the fitting results at the end of the network. This ensures more reliable results.…”

RED-Net: A Recurrent Encoder–Decoder Network for Video-Based Face Alignment

“…al. [40,37] proposed to learn a person specific model using incremental learning. However, incremental learning (or online learning) is a challenging problem, as the incremental scheme has to be carefully designed to prevent model drifting.…”

“…Despite the long history of research in rigid and non-rigid face tracking [5,34,11,35], current efforts have mostly fo-cused on face alignment in still images [43,59,50,61]. When videos are considered as input, most methods perform landmark detection by independently applying models trained on still images in each frame in a tracking-by-detection manner [53], with notable exceptions such as [2,40,37], which explore incremental learning based on previous frames. These methods do not take full advantage of the temporal information to predict face landmarks for each frame.…”

Toward Marker-Free 3D Pose Estimation in Lifting: A Deep Multi-View Solution

“…Then, raters estimate the body pose by making an optimal fit of a predefined digital manikin to the selected video frames. Finally, using the estimated body pose data and time information extracted from the videos [4], joints trajectory is generated for the entire task by applying a motion pattern prediction algorithm [5]. Observational systems are not as accurate as direct measurement systems and the result accuracy rely on the experience of the rater, especially when joints angle become close to the posture boundaries [6].…”