Deep Encoded Linguistic and Acoustic Cues for Attention Based End to End Speech Emotion Recognition

Bhosale, Swapnil; Chakraborty, Rupayan; Kopparapu, Sunil Kumar

doi:10.1109/icassp40776.2020.9054621

Cited by 20 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This development can be attributed to the fast pace of solutions to visual task recognition [14]. In [15], the authors introduced a comprehensive model incorporating convolutional layers and a multi-head selfattention mechanism, which utilized deep encoded linguistic information and audio spectrogram representation to perform emotion recognition in speech. Also, to address the class imbalance in their dataset, they carried out down-sampling and ensembling, further improving the SER accuracy.…”

Section: Related Workmentioning

confidence: 99%

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

Patamia

Acheampong

et al. 2021

2021 IEEE 2nd International Conference on Pattern Recognition and Machine Learning (PRML)

View full text Add to dashboard Cite

Emotion recognition is a topic of significant interest in assistive robotics due to the need to equip robots with the ability to comprehend human behavior, facilitating their effective interaction in our society. Consequently, efficient and dependable emotion recognition systems supporting optimal human-machine communication are required. Multi-modality (including speech, audio, text, images, and videos) is typically exploited in emotion recognition tasks. Much relevant research is based on merging multiple data modalities and training deep learning models utilizing low-level data representations. However, most existing emotion databases are not large (or complex) enough to allow machine learning approaches to learn detailed representations. This paper explores modalityspecific pre-trained transformer frameworks for self-supervised learning of speech and text representations for data-efficient emotion recognition while achieving state-of-the-art performance in recognizing emotions. This model applies feature-level fusion using nonverbal cue data points from motion capture to provide multimodal speech emotion recognition. The model was trained using the publicly available IEMOCAP dataset, achieving an overall accuracy of 77.58% for four emotions, outperforming state-of-the-art approaches

show abstract

Section: Related Workmentioning

confidence: 99%

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

Patamia

Acheampong

et al. 2021

2021 IEEE 2nd International Conference on Pattern Recognition and Machine Learning (PRML)

View full text Add to dashboard Cite

show abstract

“…Therefore, it might be beneficial to allow attention mechanisms to combine different representation subspaces using queries, keys, and values. Bhosale et al [22] were the first to combine an attention mechanism with an RNN. They calculated alignment-probability matrices for input and output series in an encoder-decoder model, effectively solving machine translation issues.…”

Section: Related Workmentioning

confidence: 99%

“…In essence, given that the input matrices Q and K are of dimension d k and V is of dimension d v , the operation executes the matrix multiplication of Q and each K, scales the product by √ d k , and thereafter applies the softmax function to ascertain the weights. The output matrix is articulated as follows [22]:…”

mentioning

confidence: 99%

Prediction of Parking Space Availability Using Improved MAT-LSTM Network

Zhang,

Shang,

Yan

et al. 2024

IJGI

View full text Add to dashboard Cite

The prediction of parking space availability plays a crucial role in information systems providing parking guidance. However, controversy persists regarding the efficiency and accuracy of mainstream time series prediction methods, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In this study, a comparison was made between a temporal convolutional network (TCN) based on CNNs and a long short-term memory (LSTM) network based on RNNs to determine an appropriate baseline for predicting parking space availability. Subsequently, a multi-head attention (MAT) mechanism was incorporated into an LSTM network, attempting to improve its accuracy. Experiments were conducted on three real and two synthetic datasets. The results indicated that the TCN achieved the fastest convergence, whereas the MAT-LSTM method provided the highest average accuracy, namely 0.0330 and 1.102 × 10−6, on the real and synthetic datasets, respectively. Furthermore, the improved MAT-LSTM model accomplished an increase of up to 48% in accuracy compared with the classic LSTM model. Consequently, we concluded that RNN-based networks are better suited for predicting long-time series. In particular, the MAT-LSTM method proposed in this study holds higher application value for predicting parking space availability with a higher accuracy.

show abstract

“…Research on the transfer of Speech Emotion Recognition (SER) models to the MER regression task [10], has shown successful transfer learning for recognition of emotions from speech in English to classical music. 1 In the case of SER, the work on linguistic research has gained more importance lately: using a bag-of-audio-words approach to exploit linguistic features [14], combining speech-based and linguistic classifiers [15], or using linguistic and acoustic cues for end-to-end models [16]. The linguistic approach to emotion recognition is mainly due to the fact that different emotion adjectives will tend to have diverse meanings across cultures [17] and that translation of words might result questionable [8].…”

Section: Related Workmentioning

confidence: 99%

Language-Sensitive Music Emotion Recognition Models: are We Really There Yet?

Sebastián

Cano

Pandrea

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Our previous research showed promising results when transferring features learned from speech to train emotion recognition models for music. In this context, we implemented a denoising autoencoder as a pretraining approach to extract features from speech in two languages (English and Mandarin). From that, we performed transfer and multi-task learning to predict classes from the arousal-valence space of music emotion. We tested and analyzed intra-linguistic and cross-linguistic settings, depending on the language of speech and lyrics of the music. This paper presents additional investigation on our approach, which reveals that: (1) performing pretraining with speech in a mixture of languages yields similar results than for specific languages -the pretraining phase appears not to exploit particular language features, (2) the music in Mandarin dataset consistently results in poor classification performance -we found low agreement in annotations, and (3) novel methodologies for representation learning (Contrastive Predictive Coding) may exploit features from both languages (i.e., pretraining on a mixture of languages) and improve classification of music emotions in both languages. From this study we conclude that more research is still needed to understand what is actually being transferred in these type of contexts.

show abstract

Deep Encoded Linguistic and Acoustic Cues for Attention Based End to End Speech Emotion Recognition

Cited by 20 publications

References 14 publications

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

Prediction of Parking Space Availability Using Improved MAT-LSTM Network

Language-Sensitive Music Emotion Recognition Models: are We Really There Yet?

Contact Info

Product

Resources

About