Multimodal End-to-End Sparse Model for Emotion Recognition

Abstract

Existing works on multimodal affective computing tasks, such as emotion recognition, generally adopt a two-phase pipeline, first extracting feature representations for each single modality with hand-crafted algorithms and then performing end-to-end learning with the extracted features. However, the extracted features are fixed and cannot be further fine-tuned on different target tasks, and manually finding feature extraction algorithms does not generalize or scale well to different tasks, which can lead to sub-optimal performance. In this paper, we develop a fully end-to-end model that connects the two phases and optimizes them jointly. In addition, we restructure the current datasets to enable the fully end-to-end training. Furthermore, to reduce the computational overhead brought by the end-to-end model, we introduce a sparse cross-modal attention mechanism for the feature extraction. Experimental results show that our fully end-to-end model significantly surpasses the current state-of-theart models based on the two-phase pipeline. Moreover, by adding the sparse cross-modal attention, our model can maintain performance with around half the computation in the feature extraction part.

Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization

“…Recently, many studies have been performed on multimodal learning (Mroueh et al, 2015;Antol et al, 2015;Donahue et al, 2015;Zadeh et al, 2017;Dai et al, , 2021. However, only a few have investigated MAS.…”

Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors

“…Most of the previous studies adopt a two-phase pipeline, first extracting unimodal features and then fusing them. Dai et al (2021) considered that it may lead to suboptimal performance since the extracted unimodal features are fixed and cannot be further improved benefiting from the downstream supervisory signals. Therefore, they proposed the multimodal endto-end sparse model, which can optimize the unimodal feature extraction and multimodal feature fusion jointly.…”

Weakly-supervised Multi-task Learning for Multimodal Affect Recognition

“…For example, quite a few works focus on the fusion of modalities, such as the Tensor Fusion Network , Memory Fusion Network (Zadeh et al, 2018a), Multimodal Adaptation Gate (Rahman et al, 2020). Additionally, Multimodal Transformer (Tsai et al, 2019) was introduced to handle unaligned data, Dai et al (2020a) proposed to use emotional embeddings to enable zero-/few-shot learning for lowresource senarios, and Dai et al (2021) introduced the sparse cross-attention to improve performance and reduce computation. Despite the remarkbale progress has been made, we find that most models suffer from the small scale of data on these tasks.…”