Group-level emotion recognition using deep models on image scene, faces, and skeletons

Guo, Xin; Polanía, Luisa F.; Barner, Kenneth E.

doi:10.1145/3136755.3143017

Cited by 28 publications

(31 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Holistic (scene-level) information is shown to be the important component in group-level classification in [10,12,24]. While analyzing the cohesiveness of a group of people, it is essential to understand the environments behind the people, e.g., students in a lecture tend to have a low cohesion level, while a group people standing and protesting at a plaza probably have high cohesiveness.…”

Section: Scene Featuresmentioning

confidence: 99%

Automatic Group Cohesiveness Detection With Multi-modal Features

Zhu

Guo

Barner

et al. 2019

2019 International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

Group cohesiveness is a compelling and often studied composition in group dynamics and group performance. The enormous number of web images of groups of people can be used to develop an effective method to detect group cohesiveness. This paper introduces an automatic group cohesiveness prediction method for the 7th Emotion Recognition in the Wild (EmotiW 2019) Grand Challenge in the category of Group-based Cohesion Prediction. The task is to predict the cohesive level for a group of people in images. To tackle this problem, a hybrid network including regression models which are separately trained on face features, skeleton features, and scene features is proposed. Predicted regression values, corresponding to each feature, are fused for the final cohesive intensity. Experimental results demonstrate that the proposed hybrid network is effective and makes promising improvements. A mean squared error (MSE) of 0.444 is achieved on the testing sets which outperforms the baseline MSE of 0.5.

show abstract

Section: Scene Featuresmentioning

confidence: 99%

Automatic Group Cohesiveness Detection With Multi-modal Features

Zhu

Guo

Barner

et al. 2019

2019 International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

show abstract

“…The winning team [35] and the third team [38] utilized two streams of CNN, one for individual emotion recognition and the other for global-level emotion recognition, which are combined to get the final prediction of GER. The second team [14] developed a hybrid network that can utilize global scene features, skeleton features of the group, and also local facial features.…”

Section: Related Workmentioning

confidence: 99%

“…The state-of-the-art approaches for the GER task [14,35,38] employed a combination of top-down and bottom-up approaches. The bottom-up approach entails detecting faces from the image, extracting features and performing expression recognition from each of them, and finally combining individual predictions for group-level prediction.…”

Section: Introductionmentioning

confidence: 99%

Group-Level Emotion Recognition using Deep Models with A Four-stream Hybrid Network

Khan

Cai

et al. 2018

Proceedings of the 20th ACM International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Group-level Emotion Recognition (GER) in the wild is a challenging task gaining lots of attention. Most recent works utilized two channels of information, a channel involving only faces and a channel containing the whole image, to solve this problem. However, modeling the relationship between faces and scene in a global image remains challenging. In this paper, we proposed a novel face-location aware global network, capturing the face location information in the form of an attention heatmap to better model such relationships. We also proposed a multi-scale face network to infer the group-level emotion from individual faces, which explicitly handles high variance in image and face size, as images in the wild are collected from different sources with different resolutions. In addition, a global blurred stream was developed to explicitly learn and extract the scene-only features. Finally, we proposed a four-stream hybrid network, consisting of the face-location aware global stream, the multi-scale face stream, a global blurred stream, and a global stream, to address the GER task, and showed the effectiveness of our method in GER sub-challenge, a part of the six Emotion Recognition in the Wild (EmotiW 2018) [10] Challenge. The proposed method achieved 65.59% and 78.39% accuracy on the testing and validation sets, respectively, and is ranked the third place on the leaderboard.

show abstract

“…The authors used a large-margin softmax loss for discriminative learning. In [11], we presented a hybrid network that exploited information from whole images, faces and the skeleton representation of the subjects in the image using CNNs. The problem of group emotion recognition is challenging due to face occlusions, illumination variations, head pose variations, varied indoor and outdoor settings, and faces at different distance from the camera which may lead to low-resolution face images [23].…”

Section: Introductionmentioning

confidence: 99%

Group-Level Emotion Recognition Using Hybrid Deep Models Based on Faces, Scenes, Skeletons and Visual Attentions

Guo

Zhu

Polanía

et al. 2018

Proceedings of the 20th ACM International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

This paper presents a hybrid deep learning network submitted to the 6th Emotion Recognition in the Wild (EmotiW 2018) Grand Challenge [9], in the category of group-level emotion recognition. Advanced deep learning models trained individually on faces, scenes, skeletons and salient regions using visual attention mechanisms are fused to classify the emotion of a group of people in an image as positive, neutral or negative. Experimental results show that the proposed hybrid network achieves 78.98% and 68.08% classification accuracy on the validation and testing sets, respectively. These results outperform the baseline of 64% and 61%, and achieved the first place in the challenge.

show abstract

Group-level emotion recognition using deep models on image scene, faces, and skeletons

Cited by 28 publications

References 22 publications

Automatic Group Cohesiveness Detection With Multi-modal Features

Automatic Group Cohesiveness Detection With Multi-modal Features

Group-Level Emotion Recognition using Deep Models with A Four-stream Hybrid Network

Group-Level Emotion Recognition Using Hybrid Deep Models Based on Faces, Scenes, Skeletons and Visual Attentions

Contact Info

Product

Resources

About