Generalized zero-shot emotion recognition from body gestures

Wu, Jinting; Zhang, Yujia; Sun, Shi-Ying; Li, Qian‐Zhong; Zhao, Xiaoguang

doi:10.1007/s10489-021-02927-w

Cited by 18 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Emotion recognition has drawn increasingly intense attention from researchers in a variety of scientific fields. Human emotions can be recognized and analyzed from visual data (facial expressions [21][22][23], behavior [24], gesture [25][26][27], pose [28]), acoustic data (speech) [29], physiological signals [30][31][32][33], or even text [34]. In this section, we will focus on SOTA approaches for VER based on facial expression analysis.…”

Section: State-of-the-art Approaches For Visual Emotion Recognitionmentioning

confidence: 99%

EMOLIPS: Towards Reliable Emotional Speech Lip-Reading

Ryumin,

Ryumina,

Ivanko

2023

Mathematics

View full text Add to dashboard Cite

In this article, we present a novel approach for emotional speech lip-reading (EMOLIPS). This two-level approach to emotional speech to text recognition based on visual data processing is motivated by human perception and the recent developments in multimodal deep learning. The proposed approach uses visual speech data to determine the type of speech emotion. The speech data are then processed using one of the emotional lip-reading models trained from scratch. This essentially resolves the multi-emotional lip-reading issue associated with most real-life scenarios. We implemented these models as a combination of EMO-3DCNN-GRU architecture for emotion recognition and 3DCNN-BiLSTM architecture for automatic lip-reading. We evaluated the models on the CREMA-D and RAVDESS emotional speech corpora. In addition, this article provides a detailed review of recent advances in automated lip-reading and emotion recognition that have been developed over the last 5 years (2018–2023). In comparison to existing research, we mainly focus on the valuable progress brought with the introduction of deep learning to the field and skip the description of traditional approaches. The EMOLIPS approach significantly improves the state-of-the-art accuracy for phrase recognition due to considering emotional features of the pronounced audio-visual speech up to 91.9% and 90.9% for RAVDESS and CREMA-D, respectively. Moreover, we present an extensive experimental investigation that demonstrates how different emotions (happiness, anger, disgust, fear, sadness, and neutral), valence (positive, neutral, and negative) and binary (emotional and neutral) affect automatic lip-reading.

show abstract

Section: State-of-the-art Approaches For Visual Emotion Recognitionmentioning

confidence: 99%

EMOLIPS: Towards Reliable Emotional Speech Lip-Reading

Ryumin,

Ryumina,

Ivanko

2023

Mathematics

View full text Add to dashboard Cite

show abstract

“…High-level features can be learned in spatial, temporal or spatial-temporal dimensions through the CNN-based network [311], the LSTM-based network [310] or the CNN-LSTM based network [312]. Recently, many studies have demonstrated the advantages of effectively combining different DLbased models and the attention mechanism [313] to improve the performance of EBGR.…”

Section: Dl-based Ebgrmentioning

confidence: 99%

“…Therefore, the existing methods fail to determine which emotional state a new body gesture belongs to. In order to recognize unknown emotions from seen body gestures or to know emotions from unseen body gestures, Banerjee et al [318] and Wu et al [313] introduced the generalized zero-shot learning framework, including CNN-based feature extraction, autoencoder-based representation learning, and emotion classifier.…”

Section: Dl-based Ebgrmentioning

confidence: 99%

A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances

Wang¹,

Song²,

Tao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Affective computing conjoins the research topics of emotion recognition and sentiment analysis, and can be realized with unimodal or multimodal data, consisting primarily of physical information (e.g., text, audio, and visual) and physiological signals (e.g., EEG and ECG). Physical-based affect recognition caters to more researchers due to the availability of multiple public databases, but it is challenging to reveal one's inner emotion hidden purposefully from facial expressions, audio tones, body gestures, etc. Physiological signals can generate more precise and reliable emotional results; yet, the difficulty in acquiring these signals hinders their practical application. Besides, by fusing physical information and physiological signals, useful features of emotional states can be obtained to enhance the performance of affective computing models. While existing reviews focus on one specific aspect of affective computing, we provide a systematical survey of important components: emotion models, databases, and recent advances. Firstly, we introduce two typical emotion models followed by five kinds of commonly used databases for affective computing. Next, we survey and taxonomize state-of-the-art unimodal affect recognition and multimodal affective analysis in terms of their detailed architectures and performances. Finally, we discuss some critical aspects of affective computing and its applications and conclude this review by pointing out some of the most promising future directions, such as the establishment of benchmark database and fusion strategies. The overarching goal of this systematic review is to help academic and industrial researchers understand the recent advances as well as new developments in this fast-paced, high-impact domain.

show abstract

“…Su et al (2020) employed the features of facial expression and speech response to represent emotional states by a cell-coupled long short-term memory (LSTM) network with an L-skip fusion mechanism, achieving an optimal accuracy of 76.9%. Wu et al (2022) proposed a generalized zero-shot learning framework composed of two branches of a hierarchical prototype network and a semantic auto-encoder, which can recognize body gesture categories by semantic information, and predict emotions through gestures. As the internal signal source, physiological signals are more authentic, reliable and accessible when describing emotions (Bi et al 2022), therefore many scholars focus on emotion recognition based on physiological signals.…”

Section: Introductionmentioning

confidence: 99%

Cross-subject emotion recognition using hierarchical feature optimization and support vector machine with multi-kernel collaboration

Pan,

Tang,

Wang

et al. 2023

Physiol. Meas.

View full text Add to dashboard Cite

Objective. Due to individual differences, it is greatly challenging to realize the multiple types of emotion identification across subjects. Approach. In this research, a hierarchical feature optimization method is proposed in order to represent emotional states effectively based on peripheral physiological signals. Firstly, sparse learning combined with binary search is employed to achieve feature selection of single signals. Then an improved fast correlation-based filter is proposed to implement fusion optimization of multi-channel signal features. Aiming at overcoming the limitations of the support vector machine (SVM), which uses a single kernel function to make decisions, the multi-kernel function collaboration strategy is proposed to improve the classification performance of SVM. Main results. The effectiveness of the proposed method is verified on the DEAP dataset. Experimental results show that the proposed method presents a competitive performance for four cross-subject types of emotion identification with an accuracy of 84% (group 1) and 85.07% (group 2). Significance. The proposed model with hierarchical feature optimization and SVM with multi-kernel function collaboration demonstrates superior emotion recognition accuracy compared to state-of-the-art techniques. In addition, the analysis based on DEAP dataset composition characteristics presents a novel perspective to explore the emotion recognition issue more objectively and comprehensively.

show abstract

Generalized zero-shot emotion recognition from body gestures

Cited by 18 publications

References 36 publications

EMOLIPS: Towards Reliable Emotional Speech Lip-Reading

EMOLIPS: Towards Reliable Emotional Speech Lip-Reading

A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances

Cross-subject emotion recognition using hierarchical feature optimization and support vector machine with multi-kernel collaboration

Contact Info

Product

Resources

About