Learning deep multimodal affective features for spontaneous speech emotion recognition

Zhang, Shiqing; Xin, Tao; Chuang, Yuelong; Zhao, Xiaoming

doi:10.1016/j.specom.2020.12.009

Cited by 66 publications

(17 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast to classic machine learning, deep learning mostly deals with either raw speech signals or timefrequency representations, and often shows better performance than classic machine learning in recent advances [18], [19]. Compared to 1D raw speech signals, 2D time-frequency representations have become more popular [20]. Typical time-frequency representations include spectrograms, Mel spectrograms, log Mel spectrograms, and Mel Frequency Cepstral Coefficients (MFCCs) [21], [22], [23].…”

Section: Speech Emotion Recognitionmentioning

confidence: 99%

Robust Federated Learning Against Adversarial Attacks for Speech Emotion Recognition

Chen¹,

Sofiane²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Due to the development of machine learning and speech processing, speech emotion recognition has been a popular research topic in recent years. However, the speech data cannot be protected when it is uploaded and processed on servers in the internet-of-things applications of speech emotion recognition. Furthermore, deep neural networks have proven to be vulnerable to human-indistinguishable adversarial perturbations. The adversarial attacks generated from the perturbations may result in deep neural networks wrongly predicting the emotional states. We propose a novel federated adversarial learning framework for protecting both data and deep neural networks. The proposed framework consists of i) federated learning for data privacy, and ii) adversarial training at the training stage and randomisation at the testing stage for model robustness. The experiments show that our proposed framework can effectively protect the speech data locally and improve the model robustness against a series of adversarial attacks.

show abstract

Section: Speech Emotion Recognitionmentioning

confidence: 99%

Robust Federated Learning Against Adversarial Attacks for Speech Emotion Recognition

Chen¹,

Sofiane²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The accuracy of the proposed system was significantly better in comparison to the individual classifier. Zhang et al [ 28 ] proposed multi- CNN to learn multimodal audio features from spontaneous speech for emotion recognition. Through a fusion of different features, complementary information could be generated which significantly improved the accuracy of emotion recognition.…”

Section: Related Workmentioning

confidence: 99%

Fusing traditionally extracted features with deep learned features from the speech spectrogram for anger and stress detection using convolution neural network

Kapoor

Kumar

2022

Multimed Tools Appl

View full text Add to dashboard Cite

Stress and anger are two negative emotions that affect individuals both mentally and physically; there is a need to tackle them as soon as possible. Automated systems are highly required to monitor mental states and to detect early signs of emotional health issues. In the present work convolutional neural network is proposed for anger and stress detection using handcrafted features and deep learned features from the spectrogram. The objective of using a combined feature set is gathering information from two different representations of speech signals to obtain more prominent features and to boost the accuracy of recognition. The proposed method of emotion assessment is more computationally efficient than similar approaches used for emotion assessment. The preliminary results obtained on experimental evaluation of the proposed approach on three datasets Toronto Emotional Speech Set (TESS), Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), and Berlin Emotional Database (EMO-DB) indicate that categorical accuracy is boosted and cross-entropy loss is reduced to a considerable extent. The proposed convolutional neural network (CNN) obtains training (T) and validation (V) categorical accuracy of T = 93.7%, V = 95.6% for TESS, T = 97.5%, V = 95.6% for EMO-DB and T = 96.7%, V = 96.7% for RAVDESS dataset.

show abstract

“…In recent years, with the rapid development of neural network, its advantages in the field of time series modeling and generation have been widely concerned and applied. In the field of natural language processing, neural networks have made breakthroughs in language modeling, speech recognition, and machine translation [ 3 ]. For the field of computer vision, it has excellent performance in object recognition, visual tracking, image generation, video analysis, etc.…”

Section: Introductionmentioning

confidence: 99%

Design of Neural Network Model for Cross-Media Audio and Video Score Recognition Based on Convolutional Neural Network Model

Liu

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

In this paper, the residual convolutional neural network is used to extract the note features in the music score image to solve the problem of model degradation; then, multiscale feature fusion is used to fuse the feature information of different levels in the same feature map to enhance the feature representation ability of the model. A network composed of a bidirectional simple loop unit and a chained time series classification function is used to identify notes, parallelizing a large number of calculations, thereby speeding up the convergence speed of training, which also makes the data in the dataset no longer need to be strict with labels. Alignment also reduces the requirements on the dataset. Aiming at the problem that the existing cross-modal retrieval methods based on common subspace are insufficient for mining local consistency within modalities, a cross-modal retrieval method fused with graph convolution is proposed. The K-nearest neighbor algorithm is used to construct modal graphs for samples of different modalities, and the original features of samples from different modalities are encoded through a symmetric graph convolutional coding network and a symmetric multilayer fully connected coding network, and the encoded features are fused and input. We jointly optimize the intramodal semantic constraints and intermodal modality-invariant constraints in the common subspace to learn highly locally consistent and semantically consistent common representations for samples from different modalities. The error value of the experimental results is used to illustrate the effect of parameters such as the number of iterations and the number of neurons on the network. In order to more accurately illustrate that the generated music sequence is very similar to the original music sequence, the generated music sequence is also framed, and finally the music sequence spectrogram and spectrogram are generated. The accuracy of the experiment is illustrated by comparing the spectrogram and the spectrogram, and genre classification predictions are also performed on the generated music to show that the network can generate music of different genres.

show abstract

Learning deep multimodal affective features for spontaneous speech emotion recognition

Cited by 66 publications

References 23 publications

Robust Federated Learning Against Adversarial Attacks for Speech Emotion Recognition

Robust Federated Learning Against Adversarial Attacks for Speech Emotion Recognition

Fusing traditionally extracted features with deep learned features from the speech spectrogram for anger and stress detection using convolution neural network

Design of Neural Network Model for Cross-Media Audio and Video Score Recognition Based on Convolutional Neural Network Model

Contact Info

Product

Resources

About