A multi-label convolutional neural network approach to cross-domain action unit detection

Ghosh, Sayan; Laksana, Eugene; Scherer, Stefan; Morency, Louis–Philippe

doi:10.1109/acii.2015.7344632

Cited by 53 publications

(58 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Performance comparison with the baseline CNN on the DISFA database [20] in terms of the average F1 score and the 2AFC score. cent approaches based on CNNs [8,9,34,28], on the two benchmark databases. As shown in Table 6, the proposed OFS-CNN achieves the state-of-the-art performance of AU recognition on the two databases 2 .…”

Section: Resultsmentioning

confidence: 99%

Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition

Han¹,

Meng²,

Li³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

Recognizing facial action units (AUs) during spontaneous facial displays is a challenging problem. Most recently, Convolutional Neural Networks (CNNs) have shown promise for facial AU recognition, where predefined and fixed convolution filter sizes are employed. In order to achieve the best performance, the optimal filter size is often empirically found by conducting extensive experimental validation. Such a training process suffers from expensive training cost, especially as the network becomes deeper.This paper proposes a novel Optimized Filter Size CNN (OFS-CNN), where the filter sizes and weights of all convolutional layers are learned simultaneously from the training data along with learning convolution filters. Specifically, the filter size is defined as a continuous variable, which is optimized by minimizing the training loss. Experimental results on two AU-coded spontaneous databases have shown that the proposed OFS-CNN is capable of estimating optimal filter size for varying image resolution and outperforms traditional CNNs with the best filter size obtained by exhaustive search. The OFS-CNN also beats the CNN using multiple filter sizes and more importantly, is much more efficient during testing with the proposed forward-backward propagation algorithm.

show abstract

Section: Resultsmentioning

confidence: 99%

Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition

Han¹,

Meng²,

Li³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

show abstract

“…Spatial representation Temporal modeling AU correlation [3,23,26,27,41,43,44] × × [1,2,7,15,17,29] × × [11,12,21,22,33,35,43] × × The proposed method generative dynamic Bayesian networks (DBN) [33] was proposed to model the AU relationships and their temporal evolutions. Rather than learning, pairwise AU relations can be explicitly inferred using statistics in annotations, and then injected such relations into a multi-task learning framework to select important patches for each AU [43].…”

Section: Au Detection Methodsmentioning

confidence: 99%

“…AU-aware deep networks [21] learned representation directly from images, and then greedily picked relevant receptive fields according to a relevance measure. Ghosh et al [12] showed that a shared representation can be directly learned from input images using a multi-label CNN. However, no temporal context was involved in learning these networks.…”

Section: Au Detection Methodsmentioning

confidence: 99%

Learning Spatial and Temporal Cues for Multi-Label Facial Action Unit Detection

Chu

Torre

Cohn

2017

2017 12th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2017)

134

View full text Add to dashboard Cite

Facial action units (AUs) are essential to decode human facial expressions. Researchers have focused on training AU detectors with a variety of features and classifiers. However, several issues remain. These are spatial representation, temporal modeling, and AU correlation. Unlike most studies that tackle these issues separately, we propose a hybrid network architecture to jointly address them. Specifically, spatial representations are extracted by a Convolutional Neural Network (CNN), which, as analyzed in this paper, is able to reduce personspecific biases caused by hand-crafted features (e.g., SIFT and Gabor). To model temporal dependencies, Long Short-Term Memory (LSTMs) are stacked on top of these representations, regardless of the lengths of input videos. The outputs of CNNs and LSTMs are further aggregated into a fusion network to produce per-frame predictions of 12 AUs. Our network naturally addresses the three issues, and leads to superior performance compared to existing methods that consider these issues independently. Extensive experiments were conducted on two large spontaneous datasets, GFT and BP4D, containing more than 400,000 frames coded with 12 AUs. On both datasets, we report significant improvement over a standard multi-label CNN and feature-based state-of-the-art. Finally, we provide visualization of the learned AU models, which, to our best knowledge, reveal how machines see facial AUs for the first time.

show abstract

“…Given an expert-annotated label y ∈ {−1, 0, +1} L for L AUs (−1/+1 indicates absence/presence of an AU, and 0 missing label) and a prediction ŷ ∈ ℝ L , this multi-label network aims to minimize the multi-label cross entropy loss:

L false(boldy, \hat{y} false) = \frac{- 1}{L} \sum_{ℓ = 1}^{L} false[y_{ℓ} > 0 false] log {\hat{y}}_{ℓ} + false[y_{ℓ} < 0 false] log false(1 - {\hat{y}}_{ℓ} false),

where [ x ] is an indicator function returning 1 if x is true, and 0 otherwise. The proposed multi-label architecture is similar to [18], which takes 40 × 40 pixel images as input. However, we used 200 × 200 pixel images in order capture more detail regarding facial texture that may aid in recognizing AUs.…”

Section: Baseline Methodsmentioning

confidence: 99%

Sayette Group Formation Task (GFT) Spontaneous Facial Expression Database

Girard

Chu

Jeni

et al. 2017

2017 12th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2017)

View full text Add to dashboard Cite

Despite the important role that facial expressions play in interpersonal communication and our knowledge that interpersonal behavior is influenced by social context, no currently available facial expression database includes multiple interacting participants. The Sayette Group Formation Task (GFT) database addresses the need for well-annotated video of multiple participants during unscripted interactions. The database includes 172,800 video frames from 96 participants in 32 three-person groups. To aid in the development of automated facial expression analysis systems, GFT includes expert annotations of FACS occurrence and intensity, facial landmark tracking, and baseline results for linear SVM, deep learning, active patch learning, and personalized classification. Baseline performance is quantified and compared using identical partitioning and a variety of metrics (including means and confidence intervals). The highest performance scores were found for the deep learning and active patch learning methods. Learn more at http://osf.io/7wcyz.

show abstract

A multi-label convolutional neural network approach to cross-domain action unit detection

Cited by 53 publications

References 21 publications

Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition

Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition

Learning Spatial and Temporal Cues for Multi-Label Facial Action Unit Detection

Sayette Group Formation Task (GFT) Spontaneous Facial Expression Database

Contact Info

Product

Resources

About