Facial Expression Recognition Based on Weighted-Cluster Loss and Deep Transfer Learning Using a Highly Imbalanced Dataset

Abstract

Facial expression recognition (FER) is a challenging problem in the fields of pattern recognition and computer vision. The recent success of convolutional neural networks (CNNs) in object detection and object segmentation tasks has shown promise in building an automatic deep CNN-based FER model. However, in real-world scenarios, performance degrades dramatically owing to the great diversity of factors unrelated to facial expressions, and due to a lack of training data and an intrinsic imbalance in the existing facial emotion datasets. To tackle these problems, this paper not only applies deep transfer learning techniques, but also proposes a novel loss function called weighted-cluster loss, which is used during the fine-tuning phase. Specifically, the weighted-cluster loss function simultaneously improves the intra-class compactness and the inter-class separability by learning a class center for each emotion class. It also takes the imbalance in a facial expression dataset into account by giving each emotion class a weight based on its proportion of the total number of images. In addition, a recent, successful deep CNN architecture, pre-trained in the task of face identification with the VGGFace2 database from the Visual Geometry Group at Oxford University, is employed and fine-tuned using the proposed loss function to recognize eight basic facial emotions from the AffectNet database of facial expression, valence, and arousal computing in the wild. Experiments on an AffectNet real-world facial dataset demonstrate that our method outperforms the baseline CNN models that use either weighted-softmax loss or center loss.

Facial Expressions Recognition Via CNNCraft-net for Static RGB Images

“…Compared to State of Art methods as shown in Table 6, Our method achieves little better results than many of the existing methods for the AffectNet dataset where CNNCraft-net overpass the best results by a range of 1% to 2%, also regards the size of our proposed model is not exceeding 146 megabytes that it is considered smaller size than other State of Art models that based on ResNet or VGG such as Georgescu, et al [10], Radu Tudor, et al [11], Li, Yong, et al [13,14], Charlie, et al [15], Hua, Wentao, et al [17] and Ngo, et al [20].…”

“…Also, Ngo, et al [20] use deep transfer learning techniques by using a squeeze-and-excitation network (SENet) model SE-ResNet-50 which pretrained for using the largest dataset for human face VGGFace2 and proposes a new loss function and named weighted-cluster loss. Also W. Xiaohua, et al [21] propose a two-level attention network for facial expression recognition in a static image, the first level used to extract the position of features while the second level is a Bidirectional Recurrent Neural Network for utilizing the relation between all features between all layers.…”

“…While emotion net network proposed by Wei, Zijun, et al [18] achieved accuracy 53.43%, 45.57% for eight and seven expressions respectively with AffectNet. While Ngo, et al [20] used the transfer learning techniques and proposed a loss function called weighted-cluster where the proposed loss achieved accuracy 60.7% while the weighted -SoftMax achieved 59.72% for eight expressions recognition in the AffectNet dataset.…”

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

“…Building upon deep transfer learning techniques, facial expression recognition (FER) was addressed in [ 18 ]. The authors tackled the challenging issues of: (i) diversity of factors, which are unrelated to facial expressions (ii) the lack of training data for FER and (iii) the intrinsic imbalance in existing facial emotion datasets.…”

Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives

“…DL aims to develop end-to-end systems to reduce the dependency from hand-crafted features, pre-processing, and feature extraction techniques (Ghayoumi, 2017 ). Notably, convolutional neural networks (CNNs) have been proven to be particularly efficient in this task (Mollahosseini et al, 2017 ; Zhang, 2017 ; Refat and Azlan, 2019 ).…”