Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings

Pepino, Leonardo; Riera, Pablo; Ferrer, Luciana

doi:10.48550/arxiv.2104.03502

Cited by 17 publications

(23 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The multiplet loss improved the accuracy by around 3% on the two datasets. [19] 2020 71.61% Muppidi and Radfar [16] 2021 77.87% Mustaqeem and Kwon [17] 2020 79.50% RAVDESS Mustaqeem and Kwon [18] 2020 80.00% Seo and Kim [23] 2020 83.33% Pepino et al [21] 2021…”

Section: Results and Comparisonsmentioning

confidence: 99%

See 1 more Smart Citation

MuLER: Multiplet-Loss for Emotion Recognition

Slimi

Zrigui

Nicolas

2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

With the rise of human-machine interactions, it has become necessary for machines to better understand humans in order to respond appropriately. Hence, in order to increase communication and interaction, it would be ideal for machines to automatically detect human emotions. Speech Emotion Recognition (SER) has been the focus of a lot of studies in the past few years. However, they can be considered poor in accuracy and must be improved. In our work, we propose a new loss function that aims to encode speeches instead of classifying them directly as the majority of the existing models do. The encoding will be done in a way that utterances with the same labels will have similar encodings. The encoded speeches were tested on two datasets, and we managed to get 88.19% accuracy with the RAVDESS (Ryerson Audiovisual Database of Emotional Speech and Song) dataset and 91.66% accuracy with the RML (Ryerson Multimedia Research Lab) dataset. CCS CONCEPTS• Computing methodologies → Machine learning; Neural networks.

show abstract

Section: Results and Comparisonsmentioning

confidence: 99%

“…Finally, a fully connected layer followed by a softmax layer is used to identify emotions. Pepino et al [21] used pre-trained wav2vec, a framework for extracting representations from raw audio data. The extracted features, the eGeMAPS descriptors, and the spectrograms, are used as inputs for a shallow neural network.…”

Section: State Of the Artmentioning

confidence: 99%

MuLER: Multiplet-Loss for Emotion Recognition

Slimi

Zrigui

Nicolas

2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

show abstract

“…Features can then be devised from the internal representations of the model. W2V has shown promising performance in SER tasks [80,81]. For our purposes, we choose the same fine-tuned XLSR-Wav2Vec2 model 5 [55] as for the transcriptions.…”

Section: Audiomentioning

confidence: 99%

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Christ¹,

Amiriparian²,

Kathan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Humour is a substantial element of human affect and cognition. Its automatic understanding can facilitate a more naturalistic human-device interaction and the humanisation of artificial intelligence. Current methods of humour detection are solely based on staged data making them inadequate for 'real-world' applications. We address this deficiency by introducing the novel Passau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising of about 11 hours of recordings. The Passau-SFCH dataset is annotated for the presence of humour and its dimensions (sentiment and direction) as proposed in Martin's Humor Style Questionnaire. We conduct a series of experiments, employing pretrained Transformers, convolutional neural networks, and expert-designed features. The performance of each modality (text, audio, video) for spontaneous humour recognition is analysed and their complementarity is investigated. Our findings suggest that for the automatic analysis of humour and its sentiment, facial expressions are most promising, while humour direction can be best modelled via text-based features. The results reveal considerable differences among various subjects, highlighting the individuality of humour usage and style. Further, we observe that a decision-level fusion yields the best recognition result. Finally, we make our code publicly available at https://www.github.com/EIHW/passau-sfch. The Passau-SFCH dataset is available upon request.

show abstract

“…For acoustic modality, wav2vec2.0 embeddings (without finetune) perform best for GMFN and Bert-MAG model. According to literature (Chen and Rudnicky, 2021;Pepino et al, 2021), finetuning wav2vec2.0 can further improve model performance which might provide more effective acoustic features for future MSA research. For Visual modality, the combination of facial landmarks and action units achieves the overall best result, revealing the effectiveness of both landmarks and action units for sentiment classification.…”

Section: Model Training Modulementioning

confidence: 99%

M-SENA: An Integrated Platform for Multimodal Sentiment Analysis

Huisheng¹,

Yuan²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

M-SENA is an open-sourced platform for Multimodal Sentiment Analysis. It aims to facilitate advanced research by providing flexible toolkits, reliable benchmarks, and intuitive demonstrations. The platform features a fully modular video sentiment analysis framework consisting of data management, feature extraction, model training, and result analysis modules. In this paper, we first illustrate the overall architecture of the M-SENA platform and then introduce features of the core modules. Reliable baseline results of different modality features and MSA benchmarks are also reported. Moreover, we use model evaluation and analysis tools provided by M-SENA to present intermediate representation visualization, on-the-fly instance test, and generalization ability test results. The source code of the platform is publicly available at https: //github.com/thuiar/M-SENA.

show abstract

Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings

Cited by 17 publications

References 32 publications

MuLER: Multiplet-Loss for Emotion Recognition

MuLER: Multiplet-Loss for Emotion Recognition

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

M-SENA: An Integrated Platform for Multimodal Sentiment Analysis

Contact Info

Product

Resources

About