Group-level Cohesion Prediction using Deep Learning Models with A Multi-stream Hybrid Network

Dang, Tien X.; Kim, Soo-Hyung; Yang, Hyung-Jeong; Lee, Guee-Sang; Vo, Hung

doi:10.1145/3340555.3355715

Cited by 5 publications

(7 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This network uses three cascaded convolutional networks to improve face detection accuracy, which makes its use very common [ 18 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ]. There are other methods that also use neural networks, such as RetinaFace [ 21 , 30 ], PyramidBox [ 19 ], TinyFace [ 19 , 20 , 31 ], and the Single-Shot Scale-Invariant Face Detector (S3FD) [ 32 ]. Other methods do not use neural networks for face detection, such as the Viola–Jones algorithm [ 33 ], which uses Haar characteristics to locate the face in an image.…”

Section: Related Workmentioning

confidence: 99%

“…In [ 26 ], two ResNet models were used: ResNet-18 for small faces (size less than 48 × 48) and ResNet-34 for large faces (size larger than 48 × 48). To improve the precision in the detection of individual emotions, apart from using a Dense Convolutional Network (DenseNet201), two neural networks (Inception-ResNet-v2) can be combined, as in [ 19 ], or new blocks (e.g., excitement and comprehension blocks) can be added to a neural network, as in [ 18 , 25 ].…”

Section: Related Workmentioning

confidence: 99%

“…The detection of groups of people improves the navigation of a social robot in indoor and outdoor environments, and the detection of group emotions allows the robot to improve HRI, exhibiting acceptable social behaviour [ 13 , 14 , 15 , 16 ], as well as associating the group emotion with the scene in which the group is participating. Nevertheless, most existing studies related to detecting group emotions are based on third-person cameras [ 17 , 18 , 19 , 20 , 21 ], but their complexity makes them unsuitable for social robots with egocentric vision due to their sensory capacity.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Group Emotion Detection Based on Social Robot Perception

Quiroz

Patiño

Díaz-Amado

et al. 2022

Sensors

View full text Add to dashboard Cite

Social robotics is an emerging area that is becoming present in social spaces, by introducing autonomous social robots. Social robots offer services, perform tasks, and interact with people in such social environments, demanding more efficient and complex Human–Robot Interaction (HRI) designs. A strategy to improve HRI is to provide robots with the capacity of detecting the emotions of the people around them to plan a trajectory, modify their behaviour, and generate an appropriate interaction with people based on the analysed information. However, in social environments in which it is common to find a group of persons, new approaches are needed in order to make robots able to recognise groups of people and the emotion of the groups, which can be also associated with a scene in which the group is participating. Some existing studies are focused on detecting group cohesion and the recognition of group emotions; nevertheless, these works do not focus on performing the recognition tasks from a robocentric perspective, considering the sensory capacity of robots. In this context, a system to recognise scenes in terms of groups of people, to then detect global (prevailing) emotions in a scene, is presented. The approach proposed to visualise and recognise emotions in typical HRI is based on the face size of people recognised by the robot during its navigation (face sizes decrease when the robot moves away from a group of people). On each frame of the video stream of the visual sensor, individual emotions are recognised based on the Visual Geometry Group (VGG) neural network pre-trained to recognise faces (VGGFace); then, to detect the emotion of the frame, individual emotions are aggregated with a fusion method, and consequently, to detect global (prevalent) emotion in the scene (group of people), the emotions of its constituent frames are also aggregated. Additionally, this work proposes a strategy to create datasets with images/videos in order to validate the estimation of emotions in scenes and personal emotions. Both datasets are generated in a simulated environment based on the Robot Operating System (ROS) from videos captured by robots through their sensory capabilities. Tests are performed in two simulated environments in ROS/Gazebo: a museum and a cafeteria. Results show that the accuracy in the detection of individual emotions is 99.79% and the detection of group emotion (scene emotion) in each frame is 90.84% and 89.78% in the cafeteria and the museum scenarios, respectively.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Group Emotion Detection Based on Social Robot Perception

Quiroz

Patiño

Díaz-Amado

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…It remains an open question whether a group level mimicry feature would further improve performances. For the occasion of the EmotiW 2019 challenge [10], some studies [13,16,50] classified high and low levels of cohesion on images from a corpus of images created via web crawling of various keywords related to social events [11]. They showed how facial expressions were impacting external annotations and achieved promising results at predicting cohesion from images.…”

Section: Related Work 21 Automated Approaches To Detect Cohesionmentioning

confidence: 99%

“…Gestures are often expressed by quantifying the total movement of the hands over time. Hand positions are most often detected either by using additional sensors [32,51], or by first deriving skeletondata [16,50], using software solutions such as OpenPose [7]. Body language cues can contain additional information on the intensity and synchronicity of the conversation in a group.…”

Section: Nonverbal Features From the Visual Channel Related To Cohesionmentioning

confidence: 99%

Modeling Dynamics of Task and Social Cohesion from the Group Perspective Using Nonverbal Motion Capture-based Features

Walocha

Maman

Chétouani

et al. 2020

Companion Publication of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

show abstract

Efficient Group-Based Cohesion Prediction in Images Using Facial Descriptors

Gavrikov¹,

Savchenko²

2021

Communications in Computer and Information Science

View full text Add to dashboard Cite

In this paper we study the problem of predicting the cohesiveness and emotion of a group of people in photo. We proposed a fast approach, consisting of face detection by using MTCNN, aggregation of facial features (age, gender and embeddings) extracted by multi-task MobileNet, prediction of group cohesion and classification of emotional background using multi-output convolution neural network. Experimental study on the Group Affect Dataset from EmotiW 2019 challenge demonstrated that our approach allows to achieve an improvement of quality and even to reduce the running time of an algorithm’s work when compared to known solutions. As a result, we obtained mean squared error 0.63 for cohesion prediction, which is 0.21 lower when compared to baseline CapsNet.

show abstract

Group-level Cohesion Prediction using Deep Learning Models with A Multi-stream Hybrid Network

Cited by 5 publications

References 20 publications

Group Emotion Detection Based on Social Robot Perception

Group Emotion Detection Based on Social Robot Perception

Modeling Dynamics of Task and Social Cohesion from the Group Perspective Using Nonverbal Motion Capture-based Features

Efficient Group-Based Cohesion Prediction in Images Using Facial Descriptors

Contact Info

Product

Resources

About