Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks

Hasani, Behzad; Mahoor, Mohammad H.

doi:10.1109/cvprw.2017.282

Cited by 223 publications

(152 citation statements)

References 64 publications

Supporting

Mentioning

151

Contrasting

Unclassified

Order By: Relevance

“…Connie et al [4] employed SIFT and dense SIFT, while Kaya et al [15] employed SIFT, HOG and Local Gabor Binary Patterns (LGBP). While most works studied facial expression recognition from static images as we do here, some works approached facial expression recognition in video [11,15].…”

Section: Related Artmentioning

confidence: 99%

Recognizing Facial Expressions of Occluded Faces Using Convolutional Neural Networks

Ionescu

2019

Communications in Computer and Information Science

View full text Add to dashboard Cite

In this paper, we present an approach based on convolutional neural networks (CNNs) for facial expression recognition in a difficult setting with severe occlusions. More specifically, our task is to recognize the facial expression of a person wearing a virtual reality (VR) headset which essentially occludes the upper part of the face. In order to accurately train neural networks for this setting, in which faces are severely occluded, we modify the training examples by intentionally occluding the upper half of the face. This forces the neural networks to focus on the lower part of the face and to obtain better accuracy rates than models trained on the entire faces. Our empirical results on two benchmark data sets, FER+ and AffectNet, show that our CNN models' predictions on lower-half faces are up to 13% higher than the baseline CNN models trained on entire faces, proving their suitability for the VR setting. Furthermore, our models' predictions on lower-half faces are no more than 10% under the baseline models' predictions on full faces, proving that there are enough clues in the lower part of the face to accurately predict facial expressions.

show abstract

Section: Related Artmentioning

confidence: 99%

Recognizing Facial Expressions of Occluded Faces Using Convolutional Neural Networks

Ionescu

2019

Communications in Computer and Information Science

View full text Add to dashboard Cite

show abstract

“…With the advent of computer vision and speech recognition, research on emotion recognition using facial expression and speech has gained prevalence [6] [7]. Hasani and Mahoor [25] proposed an enhanced neural network architecture that consists of a 3D version of the Inception-ResNet network followed by a long short-term memory (LSTM) unit for emotion recognition from facial expressions in videos. They employed four databases in classifying different emotions, including anger, fear, disgust, sadness, neutrality, contempt, happiness, and surprise.…”

Section: Emotion Recognitionmentioning

confidence: 99%

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Zheng

2019

2019 9th International IEEE/EMBS Conference on Neural Engineering (NER)

View full text Add to dashboard Cite

Compared with the rich studies on the motor brain-computer interface (BCI), the recently emerging affective BCI presents distinct challenges since the brain functional connectivity networks involving emotion are not well investigated. Previous studies on emotion recognition based on electroencephalography (EEG) signals mainly rely on single-channel-based feature extraction methods. In this paper, we propose a novel emotion-relevant critical subnetwork selection algorithm and investigate three EEG functional connectivity network features: strength, clustering coefficient, and eigenvector centrality. The discrimination ability of the EEG connectivity features in emotion recognition is evaluated on three public emotion EEG datasets: SEED, SEED-V, and DEAP. The strength feature achieves the best classification performance and outperforms the state-of-the-art differential entropy feature based on single-channel analysis for the EEG signals. The experimental results reveal that distinct functional connectivity patterns are exhibited for the five emotions of disgust, fear, sadness, happiness, and neutrality. Furthermore, we construct a multimodal emotion recognition model by combining the functional connectivity features from EEG and the features from eye movements or physiological signals using deep canonical correlation analysis. The classification accuracies of multimodal emotion recognition are 95.08 ± 6.42% on the SEED dataset, 84.51 ± 5.11% on the SEED-V dataset, and 85.34 ± 2.90% and 86.61 ± 3.76% for arousal and valence on the DEAP dataset, respectively. The results demonstrate the complementary representation properties of the EEG functional connectivity network features with eye movement data. In addition, we find that the brain networks constructed with fewer channels, i.e., 18 channels, achieve comparable performance with that of the 62-channel network with respect to multimodal emotion recognition and enable easier setups for BCI systems in real scenarios.Index Terms-Affective brain-computer interface, EEG, eye movement, brain functional connectivity network, multimodal emotion recognition. ! •

show abstract

“…Later on in [11], different combination of components both on F and the shortcut was investigated. Hasani et al [7] proposed a 3D ResNet based model for the task of facial expression recognition in which the shortcut was replaced with element-wise multiplication of the weight function ω and the input layer x l as follows:…”

Section: A Breg-netmentioning

confidence: 99%

Bounded Residual Gradient Networks (BReG-Net) for Facial Affect Computing

Hasani

Negi

Mahoor

2019

2019 14th IEEE International Conference on Automatic Face &Amp; Gesture Recognition (FG 2019)

Self Cite

View full text Add to dashboard Cite

Residual-based neural networks have shown remarkable results in various visual recognition tasks including Facial Expression Recognition (FER). Despite the tremendous efforts have been made to improve the performance of FER systems using DNNs, existing methods are not generalizable enough for practical applications. This paper introduces Bounded Residual Gradient Networks (BReG-Net) for facial expression recognition, in which the shortcut connection between the input and the output of the ResNet module is replaced with a differentiable function with a bounded gradient. This configuration prevents the network from facing the vanishing or exploding gradient problem. We show that utilizing such non-linear units will result in shallower networks with better performance. Further, by using a weighted loss function which gives a higher priority to less represented categories, we can achieve an overall better recognition rate. The results of our experiments show that BReG-Nets outperform state-of-the-art methods on three publicly available facial databases in the wild, on both the categorical and dimensional models of affect.

show abstract

Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks

Cited by 223 publications

References 64 publications

Recognizing Facial Expressions of Occluded Faces Using Convolutional Neural Networks

Recognizing Facial Expressions of Occluded Faces Using Convolutional Neural Networks

Identifying Functional Brain Connectivity Patterns for EEG-Based Emotion Recognition

Bounded Residual Gradient Networks (BReG-Net) for Facial Affect Computing

Contact Info

Product

Resources

About