Variable-state Latent Conditional Random Field models for facial expression analysis

Abstract

Automated recognition of facial expressions of emotions, and detection of facial action units (AUs), from videos depends critically on modeling of their dynamics. These dynamics are characterized by changes in temporal phases (onset-apex-offset) and intensity of emotion expressions and AUs, the appearance of which may vary considerably among target subjects, making the recognition/detection task very challenging. The state-of-the-art Latent Conditional Random Fields (L-CRF) framework allows one to efficiently encode these dynamics through the latent states accounting for the temporal consistency in emotion expression and ordinal relationships between its intensity levels, these latent states are typically assumed to be either unordered (nominal) or fully ordered (ordinal). Yet, such an approach is often too restrictive. For instance, in the case of AU detection, the goal is to discriminate between the segments of an image sequence in which this AU is active or inactive. While the sequence segments containing activation of the target AU may better be described using ordinal latent states (corresponding to the AU intensity levels), the inactive segments (i.e., where this AU does not occur) may better be described using unordered (nominal) latent states, as no assumption can be made about their underlying structure (since they can contain either neutral faces or activations of non-target AUs). To address this, we propose the variable-state L-CRF (VSL-CRF) model that automatically selects the optimal latent states for the target image sequence, based on the input data and underlying dynamics of the sequence. To reduce the model overfitting either the nominal or ordinal latent states, we propose a novel graph-Laplacian regularization of the latent states. We evaluate the VSL-CRF on the tasks of facial expression recognition using the CK+ dataset, and AU detection using the GEMEP-FERA and DISFA datasets, and show that the proposed model achieves better generalization performance compared to traditional L-CRFs and other related state-of-the-art models.

Crossing Domains for AU Coding: Perspectives, Approaches, and Measures

“…Gehrig et al [34] used kernel partial least square regression for multi-label AU detection. Walecki et al [35] proposed a variable-state Conditional Random Field model for dynamic facial expression recognition and AU detection. Valstar et al [38] applied a combination of GentleBoost, support vector machines, and hidden Markov models to encode AUs and their temporal activation models.…”

Deep Learning Model for Detection of Pain Intensity from Facial Expression

“…All these methods focus solely on feature extraction while the network output remains unstructured. Walecki et al [16] introduced model structures and esti-mate complex feature representations simultaneously by combining conditional random field (CRF) encoded AU dependencies with deep learning. They de-signed a novel Copula CNN deep learning approach for modelling multivariate ordinal variables.…”

“…The VSL-CRF also performs integration over the latent variable m, the state of which (ordinal or nominal) defines the type of the latent states for each se-quence of facial expressions [16]. The definition of the VSL-CRF in Eq.…”

“…The definition of the VSL-CRF in Eq. 5 allows it to simultaneously fit both ordinal and nominal latent states to each sequence, which may result in the model overfitting [16].…”

Comparing Methods for Assessment of Facial Dynamics in Patients with Major Neurocognitive Disorders

“…While expression recognition has attracted significant research attention [28,19,38], facial behavior analysis from naturalistic videos, associated to illumination changes, partial occlusions, pose variation, as well as low-intensity expressions pose challenges for current existing methods. In addition, while many areas of computer vision have experienced significant advancements with deep neural networks, analysis of facial dynamics has only recently benefited from deep convolutional networks [34,14,43,26].…”