Spatio-Temporal Facial Expression Recognition Using Convolutional Neural Networks and Conditional Random Fields

Abstract

Automated Facial Expression Recognition (FER) has been a challenging task for decades. Many of the existing works use hand-crafted features such as LBP, HOG, LPQ, and Histogram of Optical Flow (HOF) combined with classifiers such as Support Vector Machines for expression recognition. These methods often require rigorous hyperparameter tuning to achieve good results. Recently Deep Neural Networks (DNN) have shown to outperform traditional methods in visual object recognition. In this paper, we propose a two-part network consisting of a DNN-based architecture followed by a Conditional Random Field (CRF) module for facial expression recognition in videos. The first part captures the spatial relation within facial images using convolutional layers followed by three Inception-ResNet modules and two fully-connected layers. To capture the temporal relation between the image frames, we use linear chain CRF in the second part of our network. We evaluate our proposed network on three publicly available databases, viz. CK+, MMI, and FERA. Experiments are performed in subjectindependent and cross-database manners. Our experimental results show that cascading the deep network architecture with the CRF module considerably increases the recognition of facial expressions in videos and in particular it outperforms the stateof-the-art methods in the cross-database experiments and yields comparable results in the subject-independent experiments.

Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks

“…While Inception and ResNet have shown remarkable results in FER [20,52], these methods do not extract the temporal relations of the input data. Therefore, we propose a 3D Inception-ResNet architecture to address this issue.…”

“…Extracting these temporal relations has been studied using traditional methods in the past. Examples of these attempts are Hidden Markov Models [5,60,64] (which combine temporal information and apply segmentation on videos), Spatio-Temporal Hidden Markov Models (ST-HMM) by coupling S-HMM and T-HMM [50], Dynamic Bayesian Networks (DBN) [45,63] associated with a multi-sensory information fusion strategy, Bayesian temporal models [46] to capture the dynamic facial expression transition, and Conditional Random Fields (CRFs) [19,20,25,48] and their extensions such as Latent-Dynamic Conditional Random Fields (LD-CRFs) and Hidden Conditional Random Fields (HCRFs) [58].…”

“…In this case, we randomly se-lect 20 percent of the subjects as the test set and report the results on those subjects. Table 1 also provides the recognition rates of the traditional 2D Inception-ResNet from [20] which does not contain facial landmarks and the LSTM unit (DISFA is not experimented in this study).…”

Facial Affect Estimation in the Wild Using Deep Residual and Convolutional Networks

“…Inception and ResNet have shown remarkable results in various tasks [14,21,30,34]. For the Affect-in-the-Wild challenge, we proposed Inception-ResNet based architectures followed by LSTM units (submission 3) for the task of affect estimation.…”

Deep Facial Expression Recognition: A Survey

“…Face alignment is a traditional pre-processing step in many facerelated recognition tasks. We list some well-known approaches [16], [63] 3000 fps [64] 68 [55] Incremental [65] 49 [66] Deep learning cascaded CNN [67] 5 fast good/ very good [68] MTCNN [69] 5 [70], [71] and publicly available implementations that are widely used in deep FER. Given a series of training data, the first step is to detect the face and then to remove background and non-face areas.…”