D2C-Based Hybrid Network for Predicting Group Cohesion Scores

Abstract

Group cohesiveness represents the bonding between members in a group. Indeed, a group with high cohesiveness may easily reach success in their task. Therefore, the most critical element that affects the success of a group is group cohesiveness, which is estimated by Group Cohesion Score (GCS). This study proposed an automatic GCS estimation system for the 7 th Emotion Recognition in the Wild (EmotiW 2019) challenge in the task of the Group Cohesion Prediction. We proposed a multi-stream hybrid network based on scene-level, skeleton-level, UV coordinates-level, mid-fusion, and face-level, followed by late-fusion to combine these approaches. We also developed a joint training method called Discrete labels to Continuous scores (D2C), where discrete labels (categorical labels) directly participate in generating continuous scores. Our proposed method achieved 0.416 mean squared error on the testing set of the EmotiW 2019 dataset and became a state-of-the-art in this challenge. Furthermore, to confirm the ability of the proposed D2C method, we performed experiments on the AffectNet database and obtained relatively better results than state-of-the-art approaches.

Group Emotion Detection Based on Social Robot Perception

“…This network uses three cascaded convolutional networks to improve face detection accuracy, which makes its use very common [ 18 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ]. There are other methods that also use neural networks, such as RetinaFace [ 21 , 30 ], PyramidBox [ 19 ], TinyFace [ 19 , 20 , 31 ], and the Single-Shot Scale-Invariant Face Detector (S3FD) [ 32 ]. Other methods do not use neural networks for face detection, such as the Viola–Jones algorithm [ 33 ], which uses Haar characteristics to locate the face in an image.…”

“…To improve the feature extraction process, VGGFace can be used in conjunction with other architectures, such as Squeeze-and-Excitation Networks (SENets), Residual Networks (ResNets), Deep Convolutional Neural Networks (DCNNs), and Graph Neural Networks (GNNs), to improve accuracy in estimating individual emotions, as followed in [ 18 , 25 , 28 , 35 ]. There is another version of this neural network, known as VGG2-Senet-ft-FACE (pre-trained with the VGGFace2 database), which results from the combination of the ResNet and SENet networks, as described in [ 18 ], but it can also be used separately, as the study in [ 20 , 25 ] proposed.…”

“…In [ 30 ], to determine the emotion in videos, the authors used a Frame Attention Network (FAN), composed by a function incorporation module and a frame attention module, to generate a single feature vector. In [ 20 ], the Discrete label to Continuous score (D2C) method was implemented to estimate group cohesion scores considering the interaction between continuous and discrete labels. In [ 21 ], a different fusion method was proposed, called Non-Volume Preserving Fusion (NVPF).…”

“…The detection of groups of people improves the navigation of a social robot in indoor and outdoor environments, and the detection of group emotions allows the robot to improve HRI, exhibiting acceptable social behaviour [ 13 , 14 , 15 , 16 ], as well as associating the group emotion with the scene in which the group is participating. Nevertheless, most existing studies related to detecting group emotions are based on third-person cameras [ 17 , 18 , 19 , 20 , 21 ], but their complexity makes them unsuitable for social robots with egocentric vision due to their sensory capacity.…”