A Novel Emotion-Aware Method Based on the Fusion of Textual Description of Speech, Body Movements, and Facial Expressions

Du, Guanglong; Zeng, Yuwen; Su, Kang; Li, Chunquan; Wang, Xueqian; Teng, Shaohua; Li, Di; Liu, Peter

doi:10.1109/tim.2022.3204940

Cited by 4 publications

(1 citation statement)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MDF utilizes Euclidean distance measure to extract geometric features (3D joint position and distance), whereas MAF calculates angular features per joint with the rest of the joints under consideration. Thus, by concatenating the results of MDF and MAF, a feature vector of length 280 is obtained and inputted into the deep learning based bidirectional long short-term memory (Bi-LSTM) autoencoder framework [37] as elaborated in Figure 5. The BiLSTM framework, an enhanced version of simple LSTM, has shown its effectiveness when dealing with time-series sequential data [38].…”

Section: Feature Computationmentioning

confidence: 99%

A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions

Razzaq

Hussain

Bang

et al. 2023

Sensors

View full text Add to dashboard Cite

Multimodal emotion recognition has gained much traction in the field of affective computing, human–computer interaction (HCI), artificial intelligence (AI), and user experience (UX). There is growing demand to automate analysis of user emotion towards HCI, AI, and UX evaluation applications for providing affective services. Emotions are increasingly being used, obtained through the videos, audio, text or physiological signals. This has led to process emotions from multiple modalities, usually combined through ensemble-based systems with static weights. Due to numerous limitations like missing modality data, inter-class variations, and intra-class similarities, an effective weighting scheme is thus required to improve the aforementioned discrimination between modalities. This article takes into account the importance of difference between multiple modalities and assigns dynamic weights to them by adapting a more efficient combination process with the application of generalized mixture (GM) functions. Therefore, we present a hybrid multimodal emotion recognition (H-MMER) framework using multi-view learning approach for unimodal emotion recognition and introducing multimodal feature fusion level, and decision level fusion using GM functions. In an experimental study, we evaluated the ability of our proposed framework to model a set of four different emotional states (Happiness, Neutral, Sadness, and Anger) and found that most of them can be modeled well with significantly high accuracy using GM functions. The experiment shows that the proposed framework can model emotional states with an average accuracy of 98.19% and indicates significant gain in terms of performance in contrast to traditional approaches. The overall evaluation results indicate that we can identify emotional states with high accuracy and increase the robustness of an emotion classification system required for UX measurement.

show abstract

Section: Feature Computationmentioning

confidence: 99%