Attention guided 3D CNN-LSTM model for accurate speech based emotion recognition

Atila, Orhan; Şengür, Abdulkadir

doi:10.1016/j.apacoust.2021.108260

Cited by 90 publications

(37 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Convolution is done by applying filters to the input image data, which decreases its size (Yamashita et al, 2018). An additional operation called the Rectified Linear Unit (ReLU) (Atila and Sengür, 2021) was used after every convolution operation to generate a non-linear relationship between input and output. Finally, The pooling layer is used for secondary feature extraction, retaining the main features, reducing parameters, saving computing resources, preventing over-fitting, and improving model generalization (Suarez-Paniagua and Segura-Bedmar, 2018).…”

Section: Creating the Feature Mapsmentioning

confidence: 99%

A Novel Computational Framework for Precision Diagnosis and Subtype Discovery of Plant With Lesion

et al. 2022

View full text Add to dashboard Cite

Plants are often attacked by various pathogens during their growth, which may cause environmental pollution, food shortages, or economic losses in a certain area. Integration of high throughput phenomics data and computer vision (CV) provides a great opportunity to realize plant disease diagnosis in the early stage and uncover the subtype or stage patterns in the disease progression. In this study, we proposed a novel computational framework for plant disease identification and subtype discovery through a deep-embedding image-clustering strategy, Weighted Distance Metric and the t-stochastic neighbor embedding algorithm (WDM-tSNE). To verify the effectiveness, we applied our method on four public datasets of images. The results demonstrated that the newly developed tool is capable of identifying the plant disease and further uncover the underlying subtypes associated with pathogenic resistance. In summary, the current framework provides great clustering performance for the root or leave images of diseased plants with pronounced disease spots or symptoms.

show abstract

Section: Creating the Feature Mapsmentioning

confidence: 99%

A Novel Computational Framework for Precision Diagnosis and Subtype Discovery of Plant With Lesion

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Their proposal obtained an accuracy of 71.61% for RAVDESS. Other works as proposed in [42][43][44], also employed CNNs, MLPs, or LSTMs to solve the emotion recognition task on RAVDESS using spectrograms or preprocessed features to feed these models, obtaining accuracies of 80.00%, 96.18%, and 81%, respectively.…”

Section: Speech Emotion Recognitionmentioning

confidence: 99%

“…A major concern of the investigation of this paper is that despite the increasing number of publications that use RAVDESS, they lack a common evaluation framework, which makes it difficult to compare contributions. For example, Atila et al [43] achieved 96.1% accuracy, and although they used a 10-CV evaluation, they did not specify how they distributed the users in their folds, not being clear whether in each fold the same user takes part of the training and the test set or not, which is crucial information to replicate their setup for comparing proposals. Another example of a different setup appears in Pepino et al [40], where they used 20 users for training, two for validation, and two for the test set, classifying only seven of the eight emotions that the dataset has.…”

Section: Speech Emotion Recognitionmentioning

confidence: 99%

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

Jiménez

Griol

Callejas

et al. 2021

Sensors

View full text Add to dashboard Cite

Emotion Recognition is attracting the attention of the research community due to the multiple areas where it can be applied, such as in healthcare or in road safety systems. In this paper, we propose a multimodal emotion recognition system that relies on speech and facial information. For the speech-based modality, we evaluated several transfer-learning techniques, more specifically, embedding extraction and Fine-Tuning. The best accuracy results were achieved when we fine-tuned the CNN-14 of the PANNs framework, confirming that the training was more robust when it did not start from scratch and the tasks were similar. Regarding the facial emotion recognizers, we propose a framework that consists of a pre-trained Spatial Transformer Network on saliency maps and facial images followed by a bi-LSTM with an attention mechanism. The error analysis reported that the frame-based systems could present some problems when they were used directly to solve a video-based task despite the domain adaptation, which opens a new line of research to discover new ways to correct this mismatch and take advantage of the embedded knowledge of these pre-trained models. Finally, from the combination of these two modalities with a late fusion strategy, we achieved 80.08% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. The results revealed that these modalities carry relevant information to detect users’ emotional state and their combination enables improvement of system performance.

show abstract

“…Their proposal reached an accuracy of 71.61% for RAVDESS. Other works such as those proposed in [36][37][38] also employed CNNs, MLPs, or LSTMs to solve emotion recognition on RAVDESS using spectrograms or pre-processed features, obtaining accuracies of 80.00%, 96.18%, and 81%, respectively.…”

Section: Speech Emotion Recognitionmentioning

confidence: 99%

“…Although RAVDESS is appearing in a growing number of publications, there are not an standard evaluation criteria yet, making it complex to quantify and compare contributions. For example, in [37] they achieved 96.18% accuracy using a 10-CV evaluation. Nonetheless, they did not specify how they distributed users in each fold, making it unclear whether the same user participated in the training and test sets or not.…”

Section: Speech Emotion Recognitionmentioning

confidence: 99%

A Proposal for Multimodal Emotion Recognition Using Aural Transformers and Action Units on RAVDESS Dataset

et al. 2021

View full text Add to dashboard Cite

Emotion recognition is attracting the attention of the research community due to its multiple applications in different fields, such as medicine or autonomous driving. In this paper, we proposed an automatic emotion recognizer system that consisted of a speech emotion recognizer (SER) and a facial emotion recognizer (FER). For the SER, we evaluated a pre-trained xlsr-Wav2Vec2.0 transformer using two transfer-learning techniques: embedding extraction and fine-tuning. The best accuracy results were achieved when we fine-tuned the whole model by appending a multilayer perceptron on top of it, confirming that the training was more robust when it did not start from scratch and the previous knowledge of the network was similar to the task to adapt. Regarding the facial emotion recognizer, we extracted the Action Units of the videos and compared the performance between employing static models against sequential models. Results showed that sequential models beat static models by a narrow difference. Error analysis reported that the visual systems could improve with a detector of high-emotional load frames, which opened a new line of research to discover new ways to learn from videos. Finally, combining these two modalities with a late fusion strategy, we achieved 86.70% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. Results demonstrated that these modalities carried relevant information to detect users’ emotional state and their combination allowed to improve the final system performance.

show abstract

Attention guided 3D CNN-LSTM model for accurate speech based emotion recognition

Cited by 90 publications

References 44 publications

A Novel Computational Framework for Precision Diagnosis and Subtype Discovery of Plant With Lesion

A Novel Computational Framework for Precision Diagnosis and Subtype Discovery of Plant With Lesion

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

A Proposal for Multimodal Emotion Recognition Using Aural Transformers and Action Units on RAVDESS Dataset

Contact Info

Product

Resources

About