Survey of Deep Representation Learning for Speech Emotion Recognition

Latif, Siddique; Rana, Rajib; Khalifa, Sara; Jurdak, Raja; Qadir, Junaid; Schuller, Bjoern

doi:10.1109/taffc.2021.3114365

Cited by 62 publications

(28 citation statements)

References 188 publications

(195 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This allows CNNs to develop a deeper understanding of the provided input compared to typical multilayer perceptron models. CNNs have revolutionized the field of computer vision where they have been used for a variety of tasks such as classification, object detection, segmentation, and object counting [ 43 , 44 ] and they have also successfully been used for applications within the speech and other time series signal application domain [ 45 , 46 , 47 ]. In this paper, rather then using hand crafted features, a CNN has been used to perform feature extraction in order to take advantage of the spatial and temporal dependency capturing capabilities of CNNs.…”

Section: Methodsmentioning

confidence: 99%

A Deep Convolutional Neural Network-XGB for Direction and Severity Aware Fall Detection and Activity Recognition

Syed

Sierra-Sosa

Kumar

et al. 2022

Sensors

View full text Add to dashboard Cite

Activity and Fall detection have been a topic of keen interest in the field of ambient assisted living system research. Such systems make use of different sensing mechanisms to monitor human motion and aim to ascertain the activity being performed for health monitoring and other purposes. Towards this end, in addition to activity recognition, fall detection is an especially important task as falls can lead to injuries and sometimes even death. This work presents a fall detection and activity recognition system that not only considers various activities of daily living but also considers detection of falls while taking into consideration the direction and severity. Inertial Measurement Unit (accelerometer and gyroscope) data from the SisFall dataset is first windowed into non-overlapping segments of duration 3 s. After suitable data augmentation, it is then passed on to a Convolutional Neural Network (CNN) for feature extraction with an eXtreme Gradient Boosting (XGB) last stage for classification into the various output classes. The experiments show that the gradient boosted CNN performs better than other comparable techniques, achieving an unweighted average recall of 88%.

show abstract

Section: Methodsmentioning

confidence: 99%

A Deep Convolutional Neural Network-XGB for Direction and Severity Aware Fall Detection and Activity Recognition

Syed

Sierra-Sosa

Kumar

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…In this work, we implement convolutional neural network (CNN)-BLSTM-based classifiers due to their popularity in SER research [37]. It has been found that the performance of BLSTM can be improved by feeding it with a good emotional representation [38].…”

Section: Speech Emotion Classifiermentioning

confidence: 99%

Multitask Learning From Augmented Auxiliary Data for Improving Speech Emotion Recognition

Latif

Rana

Khalifa

et al. 2023

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

Despite recent advancements in speech emotion recognition (SER) models, state-of-the-art deep learning (DL) approaches face the challenge of the limited availability of annotated data. Large language models (LLMs) have revolutionised our understanding of natural language, introducing emergent properties that broaden comprehension in language, speech, and vision. This paper examines the potential of LLMs to annotate abundant speech data, aiming to enhance the state-of-the-art in SER. We evaluate this capability across various settings using publicly available speech emotion classification datasets. Leveraging ChatGPT, we experimentally demonstrate the promising role of LLMs in speech emotion data annotation. Our evaluation encompasses single-shot and few-shots scenarios, revealing performance variability in SER. Notably, we achieve improved results through data augmentation, incorporating ChatGPTannotated samples into existing datasets. Our work uncovers new frontiers in speech emotion classification, highlighting the increasing significance of LLMs in this field moving forward.

show abstract

“…They are commonly divided into behavioral and physiological modalities. Behavioral modalities includes emotion recognition from facial expressions [ 9 , 10 , 11 , 12 , 13 ], from gestures [ 14 , 15 , 16 ] and from speech [ 17 , 18 , 19 ], while physiological modalities include emotion recognition from physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), galvanic skin response (GSR), electrodermal activity (EDA) and so on [ 20 , 21 , 22 , 23 , 24 ].…”

Section: Introductionmentioning

confidence: 99%

A Survey on Physiological Signal-Based Emotion Recognition

Ahmad

Khan

2022

Bioengineering

View full text Add to dashboard Cite

Physiological signals are the most reliable form of signals for emotion recognition, as they cannot be controlled deliberately by the subject. Existing review papers on emotion recognition based on physiological signals surveyed only the regular steps involved in the workflow of emotion recognition such as pre-processing, feature extraction, and classification. While these are important steps, such steps are required for any signal processing application. Emotion recognition poses its own set of challenges that are very important to address for a robust system. Thus, to bridge the gap in the existing literature, in this paper, we review the effect of inter-subject data variance on emotion recognition, important data annotation techniques for emotion recognition and their comparison, data pre-processing techniques for each physiological signal, data splitting techniques for improving the generalization of emotion recognition models and different multimodal fusion techniques and their comparison. Finally, we discuss key challenges and future directions in this field.

show abstract

Survey of Deep Representation Learning for Speech Emotion Recognition

Cited by 62 publications

References 188 publications

A Deep Convolutional Neural Network-XGB for Direction and Severity Aware Fall Detection and Activity Recognition

A Deep Convolutional Neural Network-XGB for Direction and Severity Aware Fall Detection and Activity Recognition

Multitask Learning From Augmented Auxiliary Data for Improving Speech Emotion Recognition

A Survey on Physiological Signal-Based Emotion Recognition

Contact Info

Product

Resources

About