Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition

Abstract

Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various human emotions from heterogeneous visual, audio and text modalities. Previous methods mainly focus on projecting multiple modalities into a common latent space and learning an identical representation for all labels, which neglects the diversity of each modality and fails to capture richer semantic information for each label from different perspectives. Besides, associated relationships of modalities and labels have not been fully exploited. In this paper, we propose versaTile multi-modAl learning for multI-labeL emOtion Recognition (TAILOR), aiming to refine multi-modal representations and enhance discriminative capacity of each label. Specifically, we design an adversarial multi-modal refinement module to sufficiently explore the commonality among different modalities and strengthen the diversity of each modality. To further exploit label-modal dependence, we devise a BERT-like cross-modal encoder to gradually fuse private and common modality representations in a granularity descent way, as well as a label-guided decoder to adaptively generate a tailored representation for each label with the guidance of label semantics. In addition, we conduct experiments on the benchmark MMER dataset CMU-MOSEI in both aligned and unaligned settings, which demonstrate the superiority of TAILOR over the state-of-the-arts. Code is available at https://github.com/kniter1/TAILOR.

i-Code: An Integrative and Composable Multimodal Learning Framework

“…39.8 45.0 37.1 57.1 RAVEN 40.3 51.1 63.3 42.9 HHMPN 43.4 52.8 59.1 47.6 TAILOR (Zhang et al, 2022) 46.0 52.9 63.9 45.2 SIMM 41.8 48.4 48.2 48.6 ML-GCN 43 We list performance of models leveraging alignment information in Table 11. i-Code, w/o using the alignment information between transcripts and audio, can outperform many baseline models using that information.…”

“…i-Code, w/o using the alignment information between transcripts and audio, can outperform many baseline models using that information. (Hazarika et al, 2020) 39.8 45.0 37.1 57.1 RAVEN 40.3 51.1 63.3 42.9 MuIT (Tsai et al, 2019) 42.3 52.3 63.6 44.5 HHMPN 43.4 52.8 59.1 47.6 TAILOR (Zhang et al, 2022) 46.0 52.9 63.9 45.2 SIMM 41 41.6 51.7 58.8 46.1 MuIT (Tsai et al, 2019) 44.5 53.1 61.9 46.5 MISA (Hazarika et al, 2020) 43.0 50.9 45.3 57.1 HHMPN 45.9 55.9 60.2 49.6 SIMM 43.2 52.5 56.1 49.5 ML-GCN 41.1 50.9 54.6 47.6…”

“…We then test on a humor detection dataset, UR-FUNNY (Hasan et al, 2019). In this binary classification dataset, given a video clip with subscripts, video frames and sound, the task to predict (Hazarika et al, 2020) 39.8 45.0 37.1 57.1 RAVEN 40.3 51.1 63.3 42.9 MuIT (Tsai et al, 2019) 42.3 52.3 63.6 44.5 HHMPN 43.4 52.8 59.1 47.6 TAILOR (Zhang et al, 2022) 46.0 52.9 63.9 45.2 SIMM 41.8 48.4 48.2 48.6 ML-GCN 43 whether this clip will lead to immediate laughter. Baseline models include those that also leverage three-modality inputs, e.g., Bi-Bimodal-Fusion Network , Low-rank Matrix Fusion (LMF, Liu et al (2018)), MultiBench (Liang et al, 2021) and Tensor Fusion Network (TFN, Zadeh et al (2017)).…”