Jointly Fine-Tuning “BERT-Like” Self Supervised Models to Improve Multimodal Speech Emotion Recognition

Siriwardhana, Shamane; Reis, Andrew; Weerasekera, Rivindu; Nanayakkara, Suranga

doi:10.21437/interspeech.2020-1212

Cited by 68 publications

(52 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As a result, we achieve a state-of-the-art result on FSC and good accuracy on other datasets, SNIPS and Smartlights. We leave the extension of our method to other downstream tasks such as speech emotion recognition [27] and spoken question answering [28] as future work.…”

Section: Discussionmentioning

confidence: 99%

Two-Stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

Kim

Shin

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

End-to-end approaches open a new way for more accurate and efficient spoken language understanding (SLU) systems by alleviating the drawbacks of traditional pipeline systems. Previous works exploit textual information for an SLU model via pre-training with automatic speech recognition or finetuning with knowledge distillation. To utilize textual information more effectively, this work proposes a two-stage textual knowledge distillation method that matches utterancelevel representations and predicted logits of two modalities during pre-training and fine-tuning, sequentially. We use vq-wav2vec BERT as a speech encoder because it captures general and rich features. Furthermore, we improve the performance, especially in a low-resource scenario, with data augmentation methods by randomly masking spans of discrete audio tokens and contextualized hidden representations. Consequently, we push the state-of-the-art on the Fluent Speech Commands, achieving 99.7% test accuracy in the full dataset setting and 99.5% in the 10% subset setting. Throughout the ablation studies, we empirically verify that all used methods are crucial to the final performance, providing the best practice for spoken language understanding. Code is available at https://github.com/clovaai/textual-kd-slu.

show abstract

Section: Discussionmentioning

confidence: 99%

Two-Stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

Kim

Shin

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…[Siriwardhana et al] [63] searched the use of the pretrained "BERT-like" architecture for self-supervised learning (SSL) to both represent language and text modalities in order to recognize the multimodal language emotions. They demonstrate that a basic fusion mechanism (Shallow-Fusion) simplifies the overall structure and strengthens complex fusion mechanisms.…”

Section: A Multimodal Emotion Recognition Combining (Audiomentioning

confidence: 99%

Multimodal Emotion Recognition using Deep Learning

Abdullah

Ameen

Sadeeq

et al. 2021

JASTT

187

View full text Add to dashboard Cite

New research into human-computer interaction seeks to consider the consumer's emotional status to provide a seamless human-computer interface. This would make it possible for people to survive and be used in widespread fields, including education and medicine. Multiple techniques can be defined through human feelings, including expressions, facial images, physiological signs, and neuroimaging strategies. This paper presents a review of emotional recognition of multimodal signals using deep learning and comparing their applications based on current studies. Multimodal affective computing systems are studied alongside unimodal solutions as they offer higher accuracy of classification. Accuracy varies according to the number of emotions observed, features extracted, classification system and database consistency. Numerous theories on the methodology of emotional detection and recent emotional science address the following topics. This would encourage studies to understand better physiological signals of the current state of the science and its emotional awareness problems.

show abstract

“…In Tsai et al (2019a), the authors used multiple cross-modal Transformers and applies late fusion to obtain trimodal features, resulting in a large amount of parameters needed to retain original modality information. Other works that also used cross-modal Transformer architecture for include ; Siriwardhana et al (2020). In contrast to the existing works, our proposed graph method, with very small amount of model parameters, can aggregate information from multiple (more than 2) modalities at early stage by building edges between the corresponding modalities, allowing richer and more complex representation of the interactions to be learned.…”

Section: Related Workmentioning

confidence: 99%

MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Yang¹,

Wang²,

Yi³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Modal-Temporal Attention Graph (MTAG). MTAG is an interpretable graph-based neural model that provides a suitable framework for analyzing multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions across modalities and through time. Then, a novel graph fusion operation, called MTAG fusion, along with a dynamic pruning and read-out technique, is designed to efficiently process this modal-temporal graph and capture various interactions. By learning to focus only on the important interactions within the graph, MTAG achieves state-ofthe-art performance on multimodal sentiment analysis and emotion recognition benchmarks, while utilizing significantly fewer model parameters. 1

show abstract

Jointly Fine-Tuning “BERT-Like” Self Supervised Models to Improve Multimodal Speech Emotion Recognition

Cited by 68 publications

References 0 publications

Two-Stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

Two-Stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

Multimodal Emotion Recognition using Deep Learning

MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Contact Info

Product

Resources

About