Learning Affective Video Features for Facial Expression Recognition via Hybrid Deep Learning

Abstract

One key challenging issues of facial expression recognition (FER) in video sequences is to extract discriminative spatiotemporal video features from facial expression images in video sequences. In this paper, we propose a new method of FER in video sequences via a hybrid deep learning model. The proposed method first employs two individual deep convolutional neural networks (CNNs), including a spatial CNN processing static facial images and a temporal CN network processing optical flow images, to separately learn high-level spatial and temporal features on the divided video segments. These two CNNs are fine-tuned on target video facial expression datasets from a pre-trained CNN model. Then, the obtained segment-level spatial and temporal features are integrated into a deep fusion network built with a deep belief network (DBN) model. This deep fusion network is used to jointly learn discriminative spatiotemporal features. Finally, an average pooling is performed on the learned DBN segment-level features in a video sequence, to produce a fixed-length global video feature representation. Based on the global video feature representations, a linear support vector machine (SVM) is employed for facial expression classification tasks. The extensive experiments on three public video-based facial expression datasets, i.e., BAUM-1s, RML, and MMI, show the effectiveness of our proposed method, outperforming the state-of-the-arts. INDEX TERMS Facial expression recognition, spatio-temporal features, hybrid deep learning, deep convolutional neural networks, deep belief network.

Robust Blocking of Human Faces with Personal Information Using Artificial Deep Neural Computing

“…In this study, a convolutional neural network (CNN)-based deep learning technique [35][36][37][38], one of the significant machine learning algorithms [39][40][41][42] in the artificial intelligence field, is utilized to robustly detect only a human facial region from the extracted skin color distribution region. Generally, the CNN is one of the most popular structures in the deep learning research field due to its excellent image processing and pattern recognition performance.…”

Real Time Facial Expression Recognition System Based on Deep Learning

“…Using deep neural networks (DNN), in [4] they propose a new FER method based on a hybrid deep learning model. This model contains three deep models, the first two are CNN, one for spatial processing and the other for time processing, while the last one is a deep belief network (DBN) model.…”

“…The remaining 7% is surprisingly expressed in oral language [3]. Some of the applications of this topic include automated security, interactive robots, human-machine interaction computers, diagnosing mental diseases, games, education, entertainment, among others [4,5].…”

“…The majority of previous studies in FER were using image-based methods, which cannot capture the temporal variability in consecutive frames in video sequences. This is a problem because there is important information in frames sequence that is not taken advantage of [4].…”

Multi-Modality Emotion Recognition Model with GAT-Based Multi-Head Inter-Modality Attention

“…The performance of artificial agents would improve in human–agent interaction if they had good emotion recognition ability with appropriate conduct [ 1 , 2 , 3 ]. Centered on this topic, much research has examined facial expression recognition [ 4 , 5 ], speech emotion recognition [ 6 , 7 ], motion emotion recognition [ 8 , 9 ], and text emotion recognition [ 10 ].…”