Learning Spatial and Temporal Cues for Multi-Label Facial Action Unit Detection

Abstract

Facial action units (AUs) are essential to decode human facial expressions. Researchers have focused on training AU detectors with a variety of features and classifiers. However, several issues remain. These are spatial representation, temporal modeling, and AU correlation. Unlike most studies that tackle these issues separately, we propose a hybrid network architecture to jointly address them. Specifically, spatial representations are extracted by a Convolutional Neural Network (CNN), which, as analyzed in this paper, is able to reduce personspecific biases caused by hand-crafted features (e.g., SIFT and Gabor). To model temporal dependencies, Long Short-Term Memory (LSTMs) are stacked on top of these representations, regardless of the lengths of input videos. The outputs of CNNs and LSTMs are further aggregated into a fusion network to produce per-frame predictions of 12 AUs. Our network naturally addresses the three issues, and leads to superior performance compared to existing methods that consider these issues independently. Extensive experiments were conducted on two large spontaneous datasets, GFT and BP4D, containing more than 400,000 frames coded with 12 AUs. On both datasets, we report significant improvement over a standard multi-label CNN and feature-based state-of-the-art. Finally, we provide visualization of the learned AU models, which, to our best knowledge, reveal how machines see facial AUs for the first time.

Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition

“…In recent years, deep convolutional neural networks have been widely used in a variety of computer vision tasks and have achieved unprecedented progress (He et al 2016;Li et al 2017a;Liu et al 2018). There are also attempts to apply deep CNN to facial AU recognition (Zhao, Chu, and Zhang 2016;Li, Abtahi, and Zhu 2017;Bishay and Patras 2017;Chu, De la Torre, and Cohn 2016). Zhao et al (Zhao, Chu, and Zhang 2016) proposed a unified architecture for facial AU detection which incorporates a deep regional feature learning and multi-label learning modules.…”

“…shape or appearance features) or more effective discriminative learning methods (Valstar and Pantic 2006;Zhao et al 2015;Jiang, Valstar, and Pantic 2011). In recent years, deep convolutional neural networks have been widely used in AU recognition due to their powerful feature representation and end-to-end efficient learning scheme, and have greatly promoted the development of this field (Zhao, Chu, and Zhang 2016;Li, Abtahi, and Zhu 2017;Bishay and Patras 2017;Chu, De la Torre, and Cohn 2016). However, recent efforts based on deep convolutional neural networks are indulged in designing deeper and more complex network structures without exception, learning more robust feature representa-tions in a data-driven manner without explicitly considering and modeling the local characteristics of the facial organs and the linkage relationship between the facial muscles.…”

Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment

“…Note that LSTM-extended version of ROI [9] is not compared due to its input of a sequence of images instead of a single image. For a fair comparison, we use the results of LSVM, JPML, APL, and CPM reported in [31,3,10]. Table 1.…”

AU R-CNN: Encoding expert prior knowledge into R-CNN for action unit detection

“…We collect the F1 scores of the most popular state-of-theart approaches that used the same 3-fold protocol in Table 4 and Table 7 to compare our approaches with other methods. These techniques include a linear support vector machine (LSVM), active patch learning (APL [43]), JPML [13], a confidence-preserving machine (CPM [10]), a block-based region learning CNN (DRML [14]), an enhancing and cropping nets (EAC-net [4]), an ROI adaption net (ROI-Nets [3]), and LSTM fused with a simple CNN (CNN+LSTM [38]), an optimized filter size CNN (OFS-CNN [15]). We also conduct complete control experiments of AU R-CNN in Table 5 and Table 8, Table 3.…”