Emotion Recognition from Large-Scale Video Clips with Cross-Attention and Hybrid Feature Weighting Neural Networks

Zhou, Siwei; Wu, Xiaorui; Jiang, Fan; Huang, Qionghao; Huang, Changqin

doi:10.3390/ijerph20021400

Cited by 11 publications

(4 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wu et al [62] May 2019 EMOTIC Zhang et al [63] August 2019 EMOTIC Mittal et al [56] June 2020 EMOTIC Thuseethan et al [64] September 2021 EMOTIC Peng et al [65] December 2021 EMOTIC Wu et al [66] August 2022 EMOTIC Yang et al [67] October 2022 EMOTIC Lee et al [17] October 2019 CAER Lee et al [17] October 2019 CAER-S Gao et al [68] January 2021 CAER-S Zhao et al [69] May 2021 CAER-S Zhao et al [70] July 2021 CAER-S Le et al [18] December 2021 CAER-S Le et al [18] December 2021 NCAER-S 1 Wu et al [66] August 2022 CAER-S Chen et al [20] October 2022 CAER-S Costa et al [19] October 2022 CAER-S Zhou et al [71] January 2023 CAER-S…”

Section: Approach Available From Datasetmentioning

confidence: 99%

A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability

et al. 2023

View full text Add to dashboard Cite

Emotion recognition is the task of identifying and understanding human emotions from data. In the field of computer vision, there is a growing interest due to the wide range of possible applications in smart cities, health, marketing, and surveillance, among others. To date, several datasets have been proposed to allow techniques to be trained, validated, and finally deployed to production. However, these techniques have several limitations related to the construction of these datasets. In this work, we survey the datasets currently employed in state-of-the-art emotion recognition, to list and discuss their applicability and limitations in real-world scenarios. We propose experiments on the data to extract essential insights related to the provided visual information in each dataset and discuss how they impact the training and validation of techniques. We also investigate the presence of nonverbal cues in the datasets and propose experiments regarding their representativeness, visibility, and data quality. Among other discussions, we show that EMOTIC has more diverse context representations than CAER, however, with conflicting annotations. Finally, we discuss application scenarios and how techniques to approach them could leverage these datasets, suggesting approaches based on findings from these datasets to help guide future research and deployment. With this work we expect to provide a roadmap for upcoming research and experimentation in emotion recognition under real-world conditions.

show abstract

Section: Approach Available From Datasetmentioning

confidence: 99%

A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Instead of that when machine learning is used for stress detection the results are entirely different. Multiple studies (Li and Liu 2020;Mittal et al 2022;Zhou et al 2023;Liu et al 2023) have shown that with intervention of machine learning, stress detection took a new reform by getting a higher prediction accuracy in less and reallife scenario transforming lives of many.…”

Section: Introductionmentioning

confidence: 99%

An evolutionary and insightful graph neural network based hybrid model for deciphering tenacious stress detection of humans using facial emotion recognition: coinage to affective computing

Bansal,

Vyas

2024

SciEnggJ

View full text Add to dashboard Cite

In the realm of affective computing, facial emotion recognition plays a pivotal role in ensuring prooper stres detection in humans. Within the domain of facial emotion recognition, accurate detection of emotions from facial expressions holds immense significance. Existing methods often neglect intricate connections between various facial features, potentially leading to challenges in discerning naunced emotional states. In response to this, we introdue approach termed Graph Neural Network-based Facial Emotion Recognition. Our technique harnesses the capabilities of graph neural network (GNNs) to surmount these limitations. The process inititaes with the creation of a graph that encapsulates the similarity between individual facial expression samples. Subsequently, this graph is fed into the model for intricate feature mapping. The outputs generated by this integrated model ingeniously merge the feature insights from adjacent facial expression samples, thereby enhancing the representation of data for downstream emotion recognition tasks. The graph neural network model processed facial expression samples are then directed to an emotion adjacency matrix that discerns various emotional states. The final outcomes are the culmination of results from this classification process. By accurately defining the top samples exhibiting the most pronounced emotional cues, our method efficiency determines the prevailing emotional state.Through comprehensive experimentation involving generation of 99.89% accuracy on publicily available facial expression dataset, our results highlight this algorithms superiority. This approach showcases a substantial increase in recognition accuracy, outperforming the closest competing algorithms by a significant margin. This demonstrates the efficacy and promise of this algorithm in revolutionizing stress detection, potentially ushering in a new era of precise and nuanced stress analysis form facial expressions.

show abstract

“…Automatic emotion recognition is expected to be an effective technology for effortless communication and forms a major research hotspot in the field of artificial intelligence. Emotions can be identified from various sources of information, such as speech (voice), utterance transcript, facial expressions, and brain waves [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ]. Among these sources, speech can be used for emotion recognition in face-to-face mode as well as in remote communication via telephone or video calls.…”

Section: Introductionmentioning

confidence: 99%

Multi-Input Speech Emotion Recognition Model Using Mel Spectrogram and GeMAPS

Toyoshima

Okada

Ishimaru

et al. 2023

Sensors

View full text Add to dashboard Cite

The existing research on emotion recognition commonly uses mel spectrogram (MelSpec) and Geneva minimalistic acoustic parameter set (GeMAPS) as acoustic parameters to learn the audio features. MelSpec can represent the time-series variations of each frequency but cannot manage multiple types of audio features. On the other hand, GeMAPS can handle multiple audio features but fails to provide information on their time-series variations. Thus, this study proposes a speech emotion recognition model based on a multi-input deep neural network that simultaneously learns these two audio features. The proposed model comprises three parts, specifically, for learning MelSpec in image format, learning GeMAPS in vector format, and integrating them to predict the emotion. Additionally, a focal loss function is introduced to address the imbalanced data problem among the emotion classes. The results of the recognition experiments demonstrate weighted and unweighted accuracies of 0.6657 and 0.6149, respectively, which are higher than or comparable to those of the existing state-of-the-art methods. Overall, the proposed model significantly improves the recognition accuracy of the emotion “happiness”, which has been difficult to identify in previous studies owing to limited data. Therefore, the proposed model can effectively recognize emotions from speech and can be applied for practical purposes with future development.

show abstract

Emotion Recognition from Large-Scale Video Clips with Cross-Attention and Hybrid Feature Weighting Neural Networks

Cited by 11 publications

References 60 publications

A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability

A Survey on Datasets for Emotion Recognition from Vision: Limitations and In-the-Wild Applicability

An evolutionary and insightful graph neural network based hybrid model for deciphering tenacious stress detection of humans using facial emotion recognition: coinage to affective computing

Multi-Input Speech Emotion Recognition Model Using Mel Spectrogram and GeMAPS

Contact Info

Product

Resources

About