A new deep-learning framework for group emotion recognition

Wei, Qinglan; Zhao, Yijia; Liu, Fangzhou; Li, Liandong; He, Jun; Yu, Lejun; Sun, Bo

doi:10.1145/3136755.3143014

Cited by 30 publications

(28 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since then, research has been done on group affect analysis and benchmarked in EmotiW 2017. The winning team [35] and the third team [38] utilized two streams of CNN, one for individual emotion recognition and the other for global-level emotion recognition, which are combined to get the final prediction of GER. The second team [14] developed a hybrid network that can utilize global scene features, skeleton features of the group, and also local facial features.…”

Section: Related Workmentioning

confidence: 99%

“…A global stream (GS) was employed to recognize the group-level affect from the whole image. This stream is necessary and has been demonstrated to be effective by early studies [35,38], since the face stream only considers the face-related features while ignores the scene. Moreover, the face stream lacks information to model the spatial relationships and interactions between faces.…”

Section: A Global Streammentioning

confidence: 99%

“…The state-of-the-art approaches for the GER task [14,35,38] employed a combination of top-down and bottom-up approaches. The bottom-up approach entails detecting faces from the image, extracting features and performing expression recognition from each of them, and finally combining individual predictions for group-level prediction.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Group-Level Emotion Recognition using Deep Models with A Four-stream Hybrid Network

Khan

Cai

et al. 2018

Proceedings of the 20th ACM International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Group-level Emotion Recognition (GER) in the wild is a challenging task gaining lots of attention. Most recent works utilized two channels of information, a channel involving only faces and a channel containing the whole image, to solve this problem. However, modeling the relationship between faces and scene in a global image remains challenging. In this paper, we proposed a novel face-location aware global network, capturing the face location information in the form of an attention heatmap to better model such relationships. We also proposed a multi-scale face network to infer the group-level emotion from individual faces, which explicitly handles high variance in image and face size, as images in the wild are collected from different sources with different resolutions. In addition, a global blurred stream was developed to explicitly learn and extract the scene-only features. Finally, we proposed a four-stream hybrid network, consisting of the face-location aware global stream, the multi-scale face stream, a global blurred stream, and a global stream, to address the GER task, and showed the effectiveness of our method in GER sub-challenge, a part of the six Emotion Recognition in the Wild (EmotiW 2018) [10] Challenge. The proposed method achieved 65.59% and 78.39% accuracy on the testing and validation sets, respectively, and is ranked the third place on the leaderboard.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: A Global Streammentioning

confidence: 99%

See 1 more Smart Citation

Group-Level Emotion Recognition using Deep Models with A Four-stream Hybrid Network

Khan

Cai

et al. 2018

Proceedings of the 20th ACM International Conference on Multimodal Interaction

View full text Add to dashboard Cite

show abstract

“…Automated group emotion recognition has great economic value in areas such as smart city planning [2], public safety [3], hospital care [4], early detection of emergencies [5], and security monitoring [6]. In the past few years, the intelligent group emotion recognition has received full attention from the academic community and has been widely studied [7]- [9].…”

Section: Introductionmentioning

confidence: 99%

Group Emotion Recognition Based on Global and Local Features

et al. 2019

View full text Add to dashboard Cite

In order to improve the accuracy of group emotion recognition, a group emotion recognition model based on global scene feature and local face feature is constructed in this paper. When extracting global scene features, with the consideration of that the different size of the background objects may have different influences on the emotion recognition, the paper proposes a feature extraction algorithm for the global scene based on the fusion of multi-scale feature maps. With the consideration of the emotion propagation between different figures in the image, the paper proposes a LSTM based algorithm for fusion the face features among different figures. Experiments show that the global scene feature extraction algorithm proposed in this paper has higher accuracy than the global scene feature extraction algorithm based on standard network architecture. Besides, the facial emotion feature fusion algorithm based on LSTM has higher classification accuracy than the fusion algorithm based on average calculation and the algorithm based on voting. Besides, the group emotion recognition model proposed in this paper has an accuracy 24.38% higher than the benchmark method, 13.32% higher than the deep learning method and 14.57% higher than the deep learning method.INDEX TERMS Deep two-way long-term and short-term memory network, feature fusion, global scene features, group emotion recognition.

show abstract

“…Nonetheless, the emotion is shared across a group of people, and the environment can also help to recognize the correct emotion to some extent. Previous work on this task has focused on taking the scene and facial features of an image [18,21], as well as the pose of people and/or their faces [8]. These approaches, however, do not learn how to appropriately combine the information coming from the different faces and the global image.…”

Section: Introductionmentioning

confidence: 99%

An Attention Model for Group-Level Emotion Recognition

Gupta

Agrawal

Chauhan

et al. 2018

Proceedings of the 20th ACM International Conference on Multimodal Interaction

View full text Add to dashboard Cite

In this paper we propose a new approach for classifying the global emotion of images containing groups of people. To achieve this task, we consider two different and complementary sources of information: i) a global representation of the entire image (ii) a local representation where only faces are considered. While the global representation of the image is learned with a convolutional neural network (CNN), the local representation is obtained by merging face features through an attention mechanism. The two representations are first learned independently with two separate CNN branches and then fused through concatenation in order to obtain the final group-emotion classifier. For our submission to the EmotiW 2018 group-level emotion recognition challenge, we combine several variations of the proposed model into an ensemble, obtaining a final accuracy of 64.83% on the test set and ranking 4th among all challenge participants.

show abstract

A new deep-learning framework for group emotion recognition

Cited by 30 publications

References 11 publications

Group-Level Emotion Recognition using Deep Models with A Four-stream Hybrid Network

Group-Level Emotion Recognition using Deep Models with A Four-stream Hybrid Network

Group Emotion Recognition Based on Global and Local Features

An Attention Model for Group-Level Emotion Recognition

Contact Info

Product

Resources

About