Speech Emotion Recognition Using Deep Learning Techniques: A Review

Khalil, Ruhul Amin; Jones, Edward; Babar, Mohammad Inayatullah; Jan, Tariqullah; Zafar, Mohammad Haseeb; Alhussain, Thamer

doi:10.1109/access.2019.2936124

Cited by 484 publications

(198 citation statements)

References 108 publications

Supporting

Mentioning

144

Contrasting

Order By: Relevance

“…The SER plays an important role in the HCI, and researchers are making a variety of techniques in the current decade to make it efficient and robust for real-time applications [ 4 ]. In the past decade, it has been a challenging task to recognize the emotional facts and the expressive cues from the speech of an individual due to the lack of techniques and technologies [ 5 , 6 ]. In each era, researchers have worked to develop an efficient SER system, and they have developed several methods for preprocessing, features extraction, and classification [ 7 ].…”

Section: Introductionmentioning

confidence: 99%

“…With the development of skills and technologies, researchers have adopted artificial intelligence (AI) and deep learning (DL) approaches to enhance the way of the HCI, which includes emotion recognition in speech signals. Today, researchers have developed new techniques for efficient SER using AI and DL in this domain [ 6 ]. Many researchers have achieved great success in this era in order to make an efficient and robust SER system using certain DL applications, such as a deep belief network [ 11 ] (DBN), CNNs [ 12 ], and long short-term memory (LSTM) [ 13 , 14 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features

Tursunov

Mustaqeem

Kwon

2020

Sensors

127

View full text Add to dashboard Cite

Artificial intelligence (AI) and machine learning (ML) are employed to make systems smarter. Today, the speech emotion recognition (SER) system evaluates the emotional state of the speaker by investigating his/her speech signal. Emotion recognition is a challenging task for a machine. In addition, making it smarter so that the emotions are efficiently recognized by AI is equally challenging. The speech signal is quite hard to examine using signal processing methods because it consists of different frequencies and features that vary according to emotions, such as anger, fear, sadness, happiness, boredom, disgust, and surprise. Even though different algorithms are being developed for the SER, the success rates are very low according to the languages, the emotions, and the databases. In this paper, we propose a new lightweight effective SER model that has a low computational complexity and a high recognition accuracy. The suggested method uses the convolutional neural network (CNN) approach to learn the deep frequency features by using a plain rectangular filter with a modified pooling strategy that have more discriminative power for the SER. The proposed CNN model was trained on the extracted frequency features from the speech data and was then tested to predict the emotions. The proposed SER model was evaluated over two benchmarks, which included the interactive emotional dyadic motion capture (IEMOCAP) and the berlin emotional speech database (EMO-DB) speech datasets, and it obtained 77.01% and 92.02% recognition results. The experimental results demonstrated that the proposed CNN-based SER system can achieve a better recognition performance than the state-of-the-art SER systems.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features

Tursunov

Mustaqeem

Kwon

2020

Sensors

127

View full text Add to dashboard Cite

show abstract

“…Thus, emotion recognition is a critical factor for several domains such as human robot interaction, characterizing the level of interest on learning, measuring happiness and satisfaction, identifying the level of vigilance in road and safety, quantifying stress, and detecting patient's mental and physical states [1], [3]. Several methods have been proposed in the literature to evaluate emotions [4], [5], [6], [7]. Self-reporting may be the most straightforward approach to assess emotions and emotional behavior.…”

Section: Introductionmentioning

confidence: 99%

Discrimination of Genuine and Acted Emotional Expressions Using EEG Signal and Machine Learning

et al. 2020

View full text Add to dashboard Cite

We present here one of the first studies that attempt to differentiate between genuine and acted emotional expressions, using EEG data. We present the first EEG dataset (available here) with recordings of subjects with genuine and fake emotional expressions. We build our experimental paradigm for classification of smiles; genuine smiles, fake/acted smiles and neutral expression. We propose multiple methods to extract intrinsic features from three EEG emotional expressions; genuine, neutral, and fake/acted smile. We extracted EEG features using three time-frequency analysis methods: discrete wavelet transforms (DWT), empirical mode decomposition (EMD), and incorporating DWT into EMD (DWT-EMD) at three frequency bands. We then evaluated the proposed methods using several classifiers including, k-nearest neighbors (KNN), support vector machine (SVM), and artificial neural network (ANN). We carried out an experimental paradigm on 28-subjects underwent three types of emotional expressions, genuine, neutral and fake/acted. The results showed that incorporating DWT into EMD extracted more hidden features than sole DWT or sole EMD method. The power spectral feature extracted by DWT, EMD, and DWT-EMD showed different neural patterns across the three emotional expressions at all the frequency bands. We performed binary classification experiments and achieved acceptable accuracy reaching a maximum of 84% in all type of emotions, classifiers and bands using sole DWT or EMD. Meanwhile, a combination of DWT-EMD achieved the highest classification accuracy with ANN in classifying true emotional expressions from fake expressions in the alpha and beta bands with an average accuracy of 94.3% and 84.1%, respectively. Our results suggest combining DWT-EMD for future emotion studies and highlight the association of alpha and beta frequency bands with emotions.

show abstract

“…With the emergence of deep learning and increased computation powers, deep learning methods are being widely adopted for automatic feature learning in diverse areas, like natural language processing, image classification, and speech recognition [ 40 , 41 , 42 ]. Recently, it has been used to extract features automatically from data collected by mobile and wearable sensors, and then classify human activity.…”

Section: Related Workmentioning

confidence: 99%

Leveraging Wearable Sensors for Human Daily Activity Recognition with Stacked Denoising Autoencoders

Fan

Zhang

et al. 2020

Sensors

View full text Add to dashboard Cite

Activity recognition has received considerable attention in many research fields, such as industrial and healthcare fields. However, many researches about activity recognition have focused on static activities and dynamic activities in current literature, while, the transitional activities, such as stand-to-sit and sit-to-stand, are more difficult to recognize than both of them. Consider that it may be important in real applications. Thus, a novel framework is proposed in this paper to recognize static activities, dynamic activities, and transitional activities by utilizing stacked denoising autoencoders (SDAE), which is able to extract features automatically as a deep learning model rather than utilize manual features extracted by conventional machine learning methods. Moreover, the resampling technique (random oversampling) is used to improve problem of unbalanced samples due to relatively short duration characteristic of transitional activity. The experiment protocol is designed to collect twelve daily activities (three types) by using wearable sensors from 10 adults in smart lab of Ulster University, the experiment results show the significant performance on transitional activity recognition and achieve the overall accuracy of 94.88% on three types of activities. The results obtained by comparing with other methods and performances on other three public datasets verify the feasibility and priority of our framework. This paper also explores the effect of multiple sensors (accelerometer and gyroscope) to determine the optimal combination for activity recognition.

show abstract

Speech Emotion Recognition Using Deep Learning Techniques: A Review

Abstract: The authors therefore, acknowledge with thanks DSR for technical and financial support.

Cited by 484 publications

References 108 publications

Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features

Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features

Discrimination of Genuine and Acted Emotional Expressions Using EEG Signal and Machine Learning

Leveraging Wearable Sensors for Human Daily Activity Recognition with Stacked Denoising Autoencoders

Contact Info

Product

Resources

About