3D shape estimation in video sequences provides high precision evaluation of facial expressions

Abstract

Person independent and pose invariant estimation of facial expressions and action unit (AU) intensity estimation is important for situation analysis and for automated video annotation. We evaluated raw 2D shape data of the CK+ database, used Procrustes transformation and the multi-class SVM leave-one-out method for classification. We found close to 100% performance demonstrating the relevance and the strength of details of the shape. Precise 3D shape information was computed by means of Constrained Local Models (CLM) on video sequences. Such sequences offer the opportunity to compute a time-averaged '3D Personal Mean Shape' (PMS) from the estimated CLM shapes, which-upon subtraction-gives rise to person independent emotion estimation. On CK+ data PMS showed significant improvements over AU0 normalization; performance reached and sometimes surpassed state-ofthe-art results on emotion classification and on AU intensity estimation. 3D PMS from 3D CLM offers pose invariant emotion estimation that we studied by rendering a 3D emotional database for different poses and different subjects from the BU 4DFE database. Frontal shapes derived from CLM fits of the 3D shape were evaluated. Results demonstrate that shape estimation alone can be used for robust, high quality pose invariant emotion classification and AU intensity estimation.

Coupled Gaussian processes for pose-invariant facial expression recognition

“…Face-shape-based approaches use either 2D or 3D face-shape models to decouple rigid movements due to the change in head pose and nonrigid movements due to the changes in facial expressions. The methods proposed in [16], [17], and [18], for example, use 3D facial geometry deformation to recognize facial expressions in 3D images. These methods require high-quality capture of the facial texture and 3D geometrical data, and thus are not vastly applicable due to extensive and complex hardware requirements.…”

Automatic Facial Expression Analysis

“…Non-negative matrix factorisation was recently applied in Jeni et al 2012. The authors argue that each dimension corresponds to a different part of the face.…”

Markov Random Field Structures for Facial Action Unit Intensity Estimation

“…They have generally employed either posed examples, or have exploited databases that are not publicly available. The techniques previously employed include using the confidence values of Support Vector Machine (SVM) [1] or AdaBoost classifiers [6] as direct indication of intensity, employing multiple binary SVM classifiers to form a multiclass classifier which is trained on each intensity is a separate class [11], or using regression based techniques such as Relevance Vector Machines (RVMs) [9] and Support Vector Regressors (SVRs) [8,16,7].…”