Deep Multimodal Emotion Recognition on Human Speech: A Review

Koromilas, Panagiotis; Γιαννακόπουλος, Θεόδωρος

doi:10.3390/app11177962

Cited by 27 publications

(18 citation statements)

References 75 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another stream of research on ASRs focused on speech emotion recognition (SER) [29]. In the context of human-computer or human-human interaction applications, the challenge of identifying emotions in human speech signals is critical and extremely difficult [30]. The blockchain based IoT devices and systems have been created [31].…”

Section: Related Workmentioning

confidence: 99%

Harris Hawks Sparse Auto-Encoder Networks for Automatic Speech Recognition System

Ali¹,

Jaber

Khalil

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

Automatic speech recognition (ASR) is an effective technique that can convert human speech into text format or computer actions. ASR systems are widely used in smart appliances, smart homes, and biometric systems. Signal processing and machine learning techniques are incorporated to recognize speech. However, traditional systems have low performance due to a noisy environment. In addition to this, accents and local differences negatively affect the ASR system’s performance while analyzing speech signals. A precise speech recognition system was developed to improve the system performance to overcome these issues. This paper uses speech information from jim-schwoebel voice datasets processed by Mel-frequency cepstral coefficients (MFCCs). The MFCC algorithm extracts the valuable features that are used to recognize speech. Here, a sparse auto-encoder (SAE) neural network is used to classify the model, and the hidden Markov model (HMM) is used to decide on the speech recognition. The network performance is optimized by applying the Harris Hawks optimization (HHO) algorithm to fine-tune the network parameter. The fine-tuned network can effectively recognize speech in a noisy environment.

show abstract

Section: Related Workmentioning

confidence: 99%

Harris Hawks Sparse Auto-Encoder Networks for Automatic Speech Recognition System

Ali¹,

Jaber

Khalil

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Model-level fusion involves the fusion of latent representations obtained from different modality models at an intermediate level in order to leverage the advantages of both feature and decisionlevel fusion. Moreover, early fusion and late fusion prevent models from learning intra and inter-modality interaction dynamics [33], [34] respectively. Multimodal fusion involves the alignment of the features in different modalities explicitly or implicitly [35].…”

Section: Related Workmentioning

confidence: 99%

Deep Learning-Based Speech Emotion Recognition Using Multi-Level Fusion of Concurrent Features

2022

View full text Add to dashboard Cite

The detection and classification of emotional states in speech involves the analysis of audio signals and text transcriptions. There are complex relationships between the extracted features at different time intervals which ought to be analyzed to infer the emotions in speech. These relationships can be represented as spatial, temporal and semantic tendency features. In addition to emotional features that exist in each modality, the text modality consists of semantic and grammatical tendencies in the uttered sentences. Spatial and temporal features have been extracted sequentially in deep learning-based models using convolutional neural networks (CNN) followed by recurrent neural networks (RNN) which may not only be weak at the detection of the separate spatial-temporal feature representations but also the semantic tendencies in speech. In this paper, we propose a deep learning-based model named concurrent spatial-temporal and grammatical (CoSTGA) model that concurrently learns spatial, temporal and semantic representations in the local feature learning block (LFLB) which are fused as a latent vector to form an input to the global feature learning block (GFLB). We also investigate the performance of multi-level feature fusion compared to single-level fusion using the multi-level transformer encoder model (MLTED) that we also propose in this paper. The proposed CoSTGA model uses multi-level fusion first at the LFLB level where similar features (spatial or temporal) are separately extracted from a modality and secondly at the GFLB level where the spatial-temporal features are fused with the semantic tendency features. The proposed CoSTGA model uses a combination of dilated causal convolutions (DCC), bidirectional long short-term memory (BiLSTM), transformer encoders (TE), multi-head and self-attention mechanisms. Acoustic and lexical features were extracted from the interactive emotional dyadic motion capture (IEMOCAP) dataset. The proposed model achieves 75.50% and 75.82% of weighted and unweighted accuracy, 75.32% and 75.57% of recall and F1 score respectively. These results imply that concurrently learned spatial-temporal features with semantic tendencies learned in a multi-level approach improve the model's effectiveness and robustness.

show abstract

“…There is an entire class of solutions called multimodal-based affective human–computer interaction that enables computer systems to recognize specific affective states. Emotions in these approaches can be recognized in many ways, including those based on: Voice parameters (timbre, raised voice, speaking rate, linguistic analysis and errors made) [ 7 , 8 , 9 , 10 ]; Characteristics of writing [ 11 , 12 , 13 , 14 , 15 ]; Changes in facial expressions in specific areas of the face [ 16 , 17 , 18 , 19 , 20 ]; Gestures and posture analysis [ 21 , 22 , 23 , 24 ]; Characterization of biological signals, including but not limited to respiration, skin conductance, blood pressure, brain imaging, and brain bioelectrical signals [ 25 , 26 , 27 , 28 , 29 , 30 ]; Context—assessing the fit between the emotion and the context of expression [ 31 ]. …”

Section: Introductionmentioning

confidence: 99%

“…Voice parameters (timbre, raised voice, speaking rate, linguistic analysis and errors made) [ 7 , 8 , 9 , 10 ];…”

Section: Introductionmentioning

confidence: 99%

Machine Learning Algorithms for Detection and Classifications of Emotions in Contact Center Applications

Płaza

Trusz

Kęczkowska

et al. 2022

Sensors

View full text Add to dashboard Cite

Over the past few years, virtual assistant solutions used in Contact Center systems are gaining popularity. One of the main tasks of the virtual assistant is to recognize the intentions of the customer. It is important to note that quite often the actual intention expressed in a conversation is also directly influenced by the emotions that accompany that conversation. Unfortunately, scientific literature has not identified what specific types of emotions in Contact Center applications are relevant to the activities they perform. Therefore, the main objective of this work was to develop an Emotion Classification for Machine Detection of Affect-Tinged Conversational Contents dedicated directly to the Contact Center industry. In the conducted study, Contact Center voice and text channels were considered, taking into account the following families of emotions: anger, fear, happiness, sadness vs. affective neutrality of the statements. The obtained results confirmed the usefulness of the proposed classification—for the voice channel, the highest efficiency was obtained using the Convolutional Neural Network (accuracy, 67.5%; precision, 80.3; F1-Score, 74.5%), while for the text channel, the Support Vector Machine algorithm proved to be the most efficient (accuracy, 65.9%; precision, 58.5; F1-Score, 61.7%).

show abstract

Deep Multimodal Emotion Recognition on Human Speech: A Review

Cited by 27 publications

References 75 publications

Harris Hawks Sparse Auto-Encoder Networks for Automatic Speech Recognition System

Harris Hawks Sparse Auto-Encoder Networks for Automatic Speech Recognition System

Deep Learning-Based Speech Emotion Recognition Using Multi-Level Fusion of Concurrent Features

Machine Learning Algorithms for Detection and Classifications of Emotions in Contact Center Applications

Contact Info

Product

Resources

About