Group-level emotion recognition using transfer learning from face identification

Abstract

In this paper we describe our algorithmic approach, which was used for submissions in the fifth Emotion Recognition in the Wild (EmotiW 2017) group-level emotion recognition sub-challenge. We extracted feature vectors of detected faces using the Convolutional Neural Network trained for face identification task, rather than traditional pre-training on emotion recognition problems. In the final pipeline an ensemble of Random Forest classifiers was learned to predict emotion score using available training set. In case when the faces have not been detected, one member of our ensemble extracts features from the whole image. During our experimental study, the proposed approach showed the lowest error rate when compared to other explored techniques. In particular, we achieved 75.4% accuracy on the validation data, which is 20% higher than the handcrafted feature-based baseline. The source code using Keras framework is publicly available.

Analyzing Group-Level Emotion with Global Alignment Kernel based Approach

“…Recognition rate baseline [11] 52.97 VGG-face [55] 65.41 VGG-16 [55] 64.11 Resnet-50 [55] 62.65 Xception [55] 60.18 Facial emotion CNN [25] 69.97 VGG-19 scene [25] 67.2 Face-pretrained CNN [23] 60 InceptionV3-FC [23] 63.19 VGG16-FC [23] 66.30 Fusion of Scene and VGG-face [22] 65.0 Ensemble of classifiers [55] 66.51 Face-pretrained CNN + InceptionV3-FC [23] 70.09 Face-pretrained CNN + VGG16-FC [23] 72.38 SVM-GAK (RVLBP) 67.32 SVM-GAK (CNN) 70.67 SVM-CGAK 72.17…”

“…Comparing with multi-modal approaches listed in the lines 11-14, SVM-GAK can as well obtain the promising results. SVM-CGAK achieved higher recognition rate (72.17%) than Fusion of Scene and VGG-face (65.0%) [22], Face-pretrained CNN + InceptionV3-FC (70.09%) [55] and Ensemble of classifier (66.51%) [55], while SVM-CGAK obtained a little worse performance comparing with hybrid network (72.38%) [55]. Different from the hybrid network [55], they combined face and scene information.…”

“…The comparative results on the MultiEmoVA database are presented in Table 7, where we straightforwardly reported the baseline results of Mou et al [10]'s paper. In Experiment #1, according to classification accuracy, it is seen that SVM-CGAK achieved the significantly results on valence-level and arousal- [11], [22], [23], [25], [55]. Methods in Lines 1-10 represent sole feature is used, while those in Lines 11-14 mean the multi-modal for group-level facial expression recognition.…”

“…In the GAFF database, we compared our method with several comparative algorithms [11], [22], [23], [25], [55] in Table 8. The comparative algorithms were briefly described as follows:…”

“…• Tan et al [25] proposed two types of CNNs, namely, individual facial emotion CNN ("Facial emotion CNN"), and global image based CNN ("VGG-19 scene"). • Rassadin et al [55] extracted feature vectors of detected faces using the convolutional neural network (CNN) trained for face identification task. In the final pipeline an ensemble of random forest classifiers on four features from VGG-face, VGG-16, ResNet-50, and Xception on the detected faces, was learned to predict emotion score using available training set.…”