Leveraging recent advances in deep learning for audio-Visual emotion recognition

Schoneveld, Liam; Othmani, Alice; Abdelkawy, Hazem

doi:10.1016/j.patrec.2021.03.007

Cited by 162 publications

(61 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Recognition of human emotion recognition using audio and visual features is also studied in some previously proposed work. Recently, Schoneveld et al [11] used deep feature representations of the audio and visual modalities to improve the accuracy of the FER task. In addition, Zhou et al [12] explored audio features using speech-spectrogram and Log Mel-spectrogram and evaluated facial features with different CNNs and different emotion pretrained strategies.…”

Section: Related Workmentioning

confidence: 99%

Ad-Corre: Adaptive Correlation-Based Loss for Facial Expression Recognition in the Wild

Fard

Mahoor

2022

IEEE Access

View full text Add to dashboard Cite

Automated Facial Expression Recognition (FER) in the wild using deep neural networks is still challenging due to intra-class variations and inter-class similarities in facial images. Deep Metric Learning (DML) is among the widely used methods to deal with these issues by improving the discriminative power of the learned embedded features. This paper proposes an Adaptive Correlation (Ad-Corre) Loss to guide the network towards generating embedded feature vectors with high correlation for within-class samples and less correlation for between-class samples. Ad-Corre consists of 3 components called Feature Discriminator, Mean Discriminator, and Embedding Discriminator. We design the Feature Discriminator component to guide the network to create the embedded feature vectors to be highly correlated if they belong to a similar class, and less correlated if they belong to different classes. In addition, the Mean Discriminator component leads the network to make the mean embedded feature vectors of different classes to be less similar to each other. We use Xception network as the backbone of our model, and contrary to previous work, we propose an embedding feature space that contains k feature vectors. Then, the Embedding Discriminator component penalizes the network to generate the embedded feature vectors, which are dissimilar. We trained our model using the combination of our proposed loss functions called Ad-Corre Loss jointly with the cross-entropy loss. We achieved a very promising recognition accuracy on AffectNet, RAF-DB, and FER-2013. Our extensive experiments and ablation study indicate the power of our method to cope well with challenging FER tasks in the wild.

show abstract

Section: Related Workmentioning

confidence: 99%

Ad-Corre: Adaptive Correlation-Based Loss for Facial Expression Recognition in the Wild

Fard

Mahoor

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Nguyen et al [31] proposed a deep model of two-stream auto-encoders and LSTM to simultaneously learn compact representative features from A and V modalities for dimensional ER. Schonevald et al [40] explored knowledge distillation using teacher-student model for V modality and CNN model for A modality using spectrograms, and fused using RNNs. Deng et al [2] proposed iterative self distillation method for modeling the uncertainties in the labels in a multi-task framework.…”

Section: Related Work 21 A-v Fusion Based Emotion Recognitionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Several ER approaches have been proposed for videobased dimensional ER using convolutional neural networks (CNNs) to obtain the deep features, and recurrent neural networks (RNNs) to capture the temporal dynamics [40,43]. Deep models have also been widely explored for vocal emotion recognition, typically using spectrograms with 2D-CNNs [40,44], or raw wave forms with 1D-CNNs [43]. In most of the existing approaches [42,43] for dimensional ER, A-V fusion is performed by concatenating the deep features extracted from individual facial and vocal modalities, and fed to LSTM for predicting valence and arousal.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition

R¹,

Granger²,

Cardinal³

2021

2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)

View full text Add to dashboard Cite

Multimodal emotion recognition has recently gained much attention since it can leverage diverse and complementary relationships over multiple modalities, such as audio, visual, and biosignals. Most state-of-the-art methods for audio-visual (A-V) fusion rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities. In this paper, we focus on dimensional emotion recognition based on the fusion of facial and vocal modalities extracted from videos. Specifically, we propose a joint cross-attention model that relies on the complementary relationships to extract the salient features across A-V modalities, allowing for accurate prediction of continuous values of valence and arousal. The proposed fusion model efficiently leverages the inter-modal relationships, while reducing the heterogeneity between features. In particular, it computes cross-attention weights based on the correlation between joint feature representations, and that of individual modalities. By deploying a joint A-V feature representation into the crossattention module, the performance of our fusion module improves significantly over the vanilla cross-attention module. Experimental results 1 on the AffWild2 dataset highlight the robustness of our proposed A-V fusion model. It has achieved a concordance correlation coefficient (CCC) of 0.374 (0.663) and 0.363 (0.584) for valence and arousal, respectively, on test set (validation set). This is a significant improvement over the baseline of third challenge of Affective Behavior Analysis in-the-wild (ABAW3) competition, with a CCC of 0.180 (0.310) and 0.170 (0.170).

show abstract

“…At the same time, deep learning has shown outstanding contributions in many fields such as speech recognition and machine translation. Some scholars have achieved excellent results in the music emotion classification competition task through the indepth application of deep learning technology (Music Information Retrieval Evaluation Exchange), combined with the emotion recognition method of music audio [7][8][9] in Figure 1.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Music Emotion Recognition Method Based on the Combination of Knowledge Distillation and Transfer Learning

Tong

2022

Scientific Programming

View full text Add to dashboard Cite

The main difficulty of music emotion recognition is the lack of sufficient labeled data. Only the labeled data with unbalanced categories are used to train the emotion recognition model. Not only is accurate labeling of emotion categories costly and time-consuming, but it also requires extensive musical background for labelers At the same time, the emotion of music is often affected by many factors. Singing methods, music styles, arrangement methods, lyrics, and other factors will affect the expression of music emotions. This paper proposes a multimodal method based on the combination of knowledge distillation and music style transfer learning and verifies the effectiveness of the method on 20,000 songs. Experiments show that compared with traditional methods, such as single audio, single lyric, and single audio with multimodal lyric methods, the method proposed in this paper has significantly improved the accuracy of emotion recognition, and the generalization ability has been significantly improved.

show abstract

Leveraging recent advances in deep learning for audio-Visual emotion recognition

Cited by 162 publications

References 15 publications

Ad-Corre: Adaptive Correlation-Based Loss for Facial Expression Recognition in the Wild

Ad-Corre: Adaptive Correlation-Based Loss for Facial Expression Recognition in the Wild

Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition

Multimodal Music Emotion Recognition Method Based on the Combination of Knowledge Distillation and Transfer Learning

Contact Info

Product

Resources

About