Predicting Arousal and Valence from Waveforms and Spectrograms Using Deep Neural Networks

Yang, Zixiaofan; Hirschberg, Julia

doi:10.21437/interspeech.2018-2397

Cited by 37 publications

(15 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While these CCC scores are lower than results reported elsewhere [41] for the SEMAINE dataset (0.680 and 0.506 CCC for A and V, respectively) and the RECOLA dataset (0.692 and 0.423), we note that those datasets consist of recordings taken in carefully controlled conditions and annotated by groups of 6+ annotators, which ensured a level of uniformity unavailable in real-life, self-rated data.…”

Section: Resultscontrasting

confidence: 79%

COVID-19: Affect recognition through voice analysis during the winter lockdown in Scotland

Garcia

Haider

Luz

2021

Preprint

View full text Add to dashboard Cite

The COVID-19 pandemic has led to unprecedented restrictions in people's lifestyle which have affected their psychological wellbeing. In this context, this paper investigates the use of social signal processing techniques for remote assessment of emotions. It presents a machine learning method for affect recognition applied to recordings taken during the COVID-19 winter lockdown in Scotland (UK). This method is exclusively based on acoustic features extracted from voice recordings collected through home and mobile devices (i.e. phones, tablets), thus providing insight into the feasibility of monitoring people's psychological wellbeing remotely, automatically and at scale. The proposed model is able to predict affect with a concordance correlation coefficient of 0.4230 (using Random Forest) and 0.3354 (using Decision Trees) for arousal and valence respectively. Clinical relevance: In 2018/2019, 12% and 14% of Scottish adults reported depression and anxiety symptoms. Remote emotion recognition through home devices would support the detection of these difficulties, which are often underdiagnosed and, if untreated, may lead to temporal or chronic disability.

show abstract

Section: Resultscontrasting

confidence: 79%

COVID-19: Affect recognition through voice analysis during the winter lockdown in Scotland

Garcia

Haider

Luz

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…They managed to improve the valence prediction task using information from other modalities such as video and physiological signals. The work in [3] shows similar results on a couple of databases after extracting features from raw waveform and spectrogram using a convolutional neural network and passing them through a neural network based regressor to get the predicted arousal and valence scores. In [4], the authors employed a fuzzy inference based system and their results show a lower mean absolute error and a higher CCC in predicting arousal than valence across three different languages.…”

Section: Introductionmentioning

confidence: 70%

Multi-Modal Learning for Speech Emotion Recognition: An Analysis and Comparison of ASR Outputs with Ground Truth Transcription

Sahu

Mitra

Seneviratne³

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

In this paper we plan to leverage multi-modal learning and automated speech recognition (ASR) systems toward building a speech-only emotion recognition model. Previous studies have shown that emotion recognition models using only acoustic features do not perform satisfactorily in detecting valence level. Text analysis has been shown to be helpful for sentiment classification. We compared classification accuracies obtained from an audio-only model, a text-only model and a multi-modal system leveraging both by performing a cross-validation analysis on IEMOCAP dataset. Confusion matrices show it's the valence level detection thats being improved by incorporating textual information. In the second stage of experiments, we used two ASR application programming interfaces (APIs) to get the transcriptions. We compare the performances of multi-modal systems using the ASR transcriptions with each other and with that of one using ground truth transcription. We analyze the confusion matrices to determine the effect of using ASR transcriptions instead of ground truth ones on class-wise accuracies. We investigate the generalisability of such a model by performing a cross-corpus study.

show abstract

“…On the other hand, the recent advancements in deep learning, along with the available computational capabilities, have enabled the research community to build end-to-end systems for SER. A big advantage of such systems is that they can directly learn the features from spectrograms or raw waveforms [12,23,36,41,45], thereby obviating the need for extracting a large set of hand-crafted features [13]. Recent studies have proposed the use of convolutional neural network (CNN) models combined with long short-term memory (LSTM) built on spectrograms and raw waveforms, showing improved SER performance [19,23,24,26,35,36,46].…”

Section: Introductionmentioning

confidence: 99%

Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation

Padi¹,

Sadjadi²,

Sriram³

et al. 2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Automatic speech emotion recognition (SER) is a challenging task that plays a crucial role in natural human-computer interaction. One of the main challenges in SER is data scarcity, i.e., insufficient amounts of carefully labeled data to build and fully explore complex deep learning models for emotion classification. This paper aims to address this challenge using a transfer learning strategy combined with spectrogram augmentation. Specifically, we propose a transfer learning approach that leverages a pre-trained residual network (ResNet) model including a statistics pooling layer from speaker recognition trained using large amounts of speaker-labeled data. The statistics pooling layer enables the model to efficiently process variable-length input, thereby eliminating the need for sequence truncation which is commonly used in SER systems. In addition, we adopt a spectrogram augmentation technique to generate additional training data samples by applying random time-frequency masks to log-mel spectrograms to mitigate overfitting and improve the generalization of emotion recognition models. We evaluate the effectiveness of our proposed approach on the interactive emotional dyadic motion capture (IEMOCAP) dataset. Experimental results indicate that the transfer learning and spectrogram augmentation approaches improve the SER performance, and when combined achieve state-of-the-art results.

show abstract

Predicting Arousal and Valence from Waveforms and Spectrograms Using Deep Neural Networks

Cited by 37 publications

References 31 publications

COVID-19: Affect recognition through voice analysis during the winter lockdown in Scotland

COVID-19: Affect recognition through voice analysis during the winter lockdown in Scotland

Multi-Modal Learning for Speech Emotion Recognition: An Analysis and Comparison of ASR Outputs with Ground Truth Transcription

Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation

Contact Info

Product

Resources

About