Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

Siriwardhana, Shamane; Reis, Andrew; Weerasekera, Rivindu; Nanayakkara, Suranga

doi:10.48550/arxiv.2008.06682

Cited by 17 publications

(38 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Eventually, we examine TEASEL against networks which fine-tune a Transformer for the downstream task. Our method has outperformed Self-MM [7], MAG-BERT, MAG-XLNet [33], and Shallow-fusion [34] methods in most metrics. Unlike MAG-Transformer methods, TEASEL does not require an aligned feature and explicitly feeds speech representations to the Transformer.…”

Section: Quantitative Analysismentioning

confidence: 95%

“…[33] have proposed a method to fuse other modalities in the middle layer of a pre-trained Transformer-based language model in an aligned manner using a Multimodal Adaption Gate (MAG) module. Later, with the popularity of Transformerbased models in Speech, [34] has examined jointly finetuning lexicon and speech Transformer on the multimodal language task. They implemented Co-Attention fusions and Shallow-Fusion using an attentive and a straightforward late fusion of two BERT-style [11] Transformers, respectively.…”

Section: Human Multimodal Language Analysismentioning

confidence: 99%

“…• Shallow-Fusion method jointly fine-tunes robertalarge and vq-wav2vec models on downstream tasks using a late fusion method [34] .…”

Section: Baselinementioning

confidence: 99%

See 2 more Smart Citations

TEASEL: A Transformer-Based Speech-Prefixed Language Model

Arjmand¹,

Dousti²,

Moradi³

2021

Preprint

View full text Add to dashboard Cite

Multimodal language analysis is a burgeoning field of NLP that aims to simultaneously model a speaker's words, acoustical annotations, and facial expressions. In this area, lexicon features usually outperform other modalities because they are pre-trained on large corpora via Transformer-based models. Despite their strong performance, training a new self-supervised learning (SSL) Transformer on any modality is not usually attainable due to insufficient data, which is the case in multimodal language learning. This work proposes a Transformer-Based Speech-Prefixed Language Model called TEASEL to approach the mentioned constraints without training a complete Transformer model. TEASEL model includes speech modality as a dynamic prefix besides the textual modality compared to a conventional language model. This method exploits a conventional pre-trained language model as a cross-modal Transformer model. We evaluated TEASEL for the multimodal sentiment analysis task defined by CMU-MOSI dataset. Extensive experiments show that our model outperforms unimodal baseline language models by 4% and outperforms the current multimodal state-of-the-art (SoTA) model by 1% in F1-score. Additionally, our proposed method is 72% smaller than the SoTA model.

show abstract

Section: Quantitative Analysismentioning

confidence: 95%

Section: Human Multimodal Language Analysismentioning

confidence: 99%

See 1 more Smart Citation

TEASEL: A Transformer-Based Speech-Prefixed Language Model

Arjmand¹,

Dousti²,

Moradi³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, several models for automatic speech recognition (ASR) which use self-supervised pretraining have been released, including wav2vec [23] and VQ-wav2vec [24]. A few recent studies [25,26,27] have successfully applied representations from these models as features for emotion recognition.…”

Section: Introductionmentioning

confidence: 99%

Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings

Pepino¹,

Riera²,

Ferrer³

2021

Preprint

View full text Add to dashboard Cite

Emotion recognition datasets are relatively small, making the use of the more sophisticated deep learning approaches challenging. In this work, we propose a transfer learning method for speech emotion recognition where features extracted from pre-trained wav2vec 2.0 models are modeled using simple neural networks. We propose to combine the output of several layers from the pre-trained model using trainable weights which are learned jointly with the downstream model. Further, we compare performance using two different wav2vec 2.0 models, with and without finetuning for speech recognition. We evaluate our proposed approaches on two standard emotion databases IEMOCAP and RAVDESS, showing superior performance compared to results in the literature.

show abstract

“…feature rich yet efficient representations (Zadeh et al 2017;Liu et al 2018;Hazarika, Zimmermann, and Poria 2020). Recently (Rahman et al 2020) used pre-trained transformer (Tsai et al 2019;Siriwardhana et al 2020) based models to achieve state-of the-art results on multimodal sentiment benchmark MOSI (Wöllmer et al 2013) and MOSEI (Zadeh et al 2018c).…”

Section: Introductionmentioning

confidence: 99%

Neural Dependency Coding inspired Multimodal Fusion

Shankar

2021

Preprint

View full text Add to dashboard Cite

Information integration from different modalities is an active area of research. Human beings and, in general, biological neural systems are quite adept at using a multitude of signals from different sensory perceptive fields to interact with the environment and each other. Recent work in deep fusion models via neural networks has led to substantial improvements over unimodal approaches in areas like speech recognition, emotion recognition and analysis, captioning and image description. However, such research has mostly focused on architectural changes allowing for fusion of different modalities while keeping the model complexity manageable. Inspired by recent neuroscience ideas about multisensory integration and processing, we investigate the effect of introducing neural dependencies in the loss functions. Experiments on multimodal sentiment analysis tasks: CMU-MOSI and CMU-MOSEI with different models show that our approach provides a consistent performance boost.

show abstract

Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

Cited by 17 publications

References 33 publications

TEASEL: A Transformer-Based Speech-Prefixed Language Model

TEASEL: A Transformer-Based Speech-Prefixed Language Model

Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings

Neural Dependency Coding inspired Multimodal Fusion

Contact Info

Product

Resources

About