Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition

Zhang, Yi; Chen, Ming‐Yuan; Shen, Jundong; Wang, Chongjun

doi:10.1609/aaai.v36i8.20895

Cited by 35 publications

(12 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, Fu et al ( 2022 ) proposed a non-homogeneous fusion network by first fusing audio and visual feature sequences through an attention aggregation module and then fusing audio-visual features with textual feature sequence via cross-modal attention. Similarly, Zhang et al ( 2022 ) proposed a hierarchical cross-modal encoder module to gradually fuse the modality features. Specifically, an adversarial multimodal refinement module was designed to decompose each modality-specific features to common and private representations.…”

Section: Related Workmentioning

confidence: 99%

“…In these works (Tsai et al, 2019 ; He et al, 2021 ; Zheng et al, 2022 ), the audio, visual, and text modalities were treated as three time-series that play the same role. Several works proposed to first fuse the audio and visual feature sequences into a higher level space, then fuse this bimodal feature sequence with the textual feature sequence (Fu et al, 2022 ; Zhang et al, 2022 ). Alternatively, text-centered frameworks were designed to explore the cross-modal interactions between textual and non-textual feature sequences (Han et al, 2021 ; He and Hu, 2021 ; Wu et al, 2021 ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal interaction enhanced representation learning for video emotion recognition

2022

View full text Add to dashboard Cite

Video emotion recognition aims to infer human emotional states from the audio, visual, and text modalities. Previous approaches are centered around designing sophisticated fusion mechanisms, but usually ignore the fact that text contains global semantic information, while speech and face video show more fine-grained temporal dynamics of emotion. From the perspective of cognitive sciences, the process of emotion expression, either through facial expression or speech, is implicitly regulated by high-level semantics. Inspired by this fact, we propose a multimodal interaction enhanced representation learning framework for emotion recognition from face video, where a semantic enhancement module is first designed to guide the audio/visual encoder using the semantic information from text, then the multimodal bottleneck Transformer is adopted to further reinforce the audio and visual representations by modeling the cross-modal dynamic interactions between the two feature sequences. Experimental results on two benchmark emotion databases indicate the superiority of our proposed method. With the semantic enhanced audio and visual features, it outperforms the state-of-the-art models which fuse the features or decisions from the audio, visual and text modalities.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multimodal interaction enhanced representation learning for video emotion recognition

2022

View full text Add to dashboard Cite

show abstract

“…However, unifying multiple modalities into one identical representation can inevitably neglect the specificity of each modality, thus losing the rich discriminative features. Although recent works (Hazarika, Zimmermann, and Poria 2020;Zhang et al 2022) attempt to learn modality-specific representations, they still utilize attention to fuse these representations into one. Therefore, a key challenge of MMER is how to effectively represent multi-modal data while maintaining modality specificity and integrating complementary information.…”

Section: Introductionmentioning

confidence: 99%

CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition

Peng,

Chen,

Shou

et al. 2024

AAAI

View full text Add to dashboard Cite

Multi-modal multi-label emotion recognition (MMER) aims to identify relevant emotions from multiple modalities. The challenge of MMER is how to effectively capture discriminative features for multiple labels from heterogeneous data. Recent studies are mainly devoted to exploring various fusion strategies to integrate multi-modal information into a unified representation for all labels. However, such a learning scheme not only overlooks the specificity of each modality but also fails to capture individual discriminative features for different labels. Moreover, dependencies of labels and modalities cannot be effectively modeled. To address these issues, this paper presents ContrAstive feature Reconstruction and AggregaTion (CARAT) for the MMER task. Specifically, we devise a reconstruction-based fusion mechanism to better model fine-grained modality-to-label dependencies by contrastively learning modal-separated and label-specific features. To further exploit the modality complementarity, we introduce a shuffle-based aggregation strategy to enrich co-occurrence collaboration among labels. Experiments on two benchmark datasets CMU-MOSEI and M3ED demonstrate the effectiveness of CARAT over state-of-the-art methods. Code is available at https://github.com/chengzju/CARAT.

show abstract

“…Compared to unisensory recognition, multisensory recognition, which considers information from various senses, demonstrates superior performance. This superiority has led to extensive research in multisensory recognition tasks, resulting in significant progress in areas such as emotion recognition (Ju et al 2020;Zhang et al 2022), medical diagnosis (Boehm et al 2022a,b), and intelligent robotics (Papanastasiou et al 2019;Heredia et al 2022).…”

Section: Introductionmentioning

confidence: 99%

ND-MRM: Neuronal Diversity Inspired Multisensory Recognition Model

Wang,

Fan,

Jia

et al. 2024

AAAI

View full text Add to dashboard Cite

Cross-sensory interaction is a key aspect for multisensory recognition. Without cross-sensory interaction, artificial neural networks show inferior performance in multisensory recognition. On the contrary, the human brain has an inherently remarkable ability in multisensory recognition, which stems from the diverse neurons that exhibit distinct responses to sensory inputs, especially the multisensory neurons with multisensory responses hence enabling cross-sensory interaction. Based on this neuronal diversity, we propose a Neuronal Diversity inspired Multisensory Recognition Model (ND-MRM), which, similar to the brain, comprises unisensory neurons and multisensory neurons. To reflect the different responses characteristics of diverse neurons in the brain, special connection constraints are innovatively designed to regulate the features transmission in the ND-MRM. Leveraging this novel concept of neuronal diversity, our model is biologically plausible, enabling more effective recognition of multisensory information. To validate the performance of the proposed ND-MRM, we employ a multisensory emotion recognition task as a case study. The results demonstrate that our model surpasses state-of-the-art brain-inspired baselines on two datasets, proving the potential of brain-inspired methods for advancing multisensory interaction and recognition.

show abstract

Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition

Cited by 35 publications

References 42 publications

Multimodal interaction enhanced representation learning for video emotion recognition

Multimodal interaction enhanced representation learning for video emotion recognition

CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition

ND-MRM: Neuronal Diversity Inspired Multisensory Recognition Model

Contact Info

Product

Resources

About