Multi-attention Recurrent Network for Human Communication Comprehension

Zadeh, Amir; Liang, Paul Pu; Poria, Soujanya; Vij, Prateek; Cambria, Erik; Morency, Louis–Philippe

doi:10.1609/aaai.v32i1.12024

Cited by 326 publications

(83 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…et al (2017) propose the TFN model, which is a multi-modal method using the tensor outer product. Liang et al (2018) propose the model which use a multi-level attention mechanism to extract different modal interaction information. Cai et al (2019) propose a hierarchical fusion model to model graphic information for irony recognition.…”

Section: So We Propose a New Multi-modal Emotionalmentioning

confidence: 99%

EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition

Li¹,

Tang²,

Zhao³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

Emotion recognition in conversation (ERC) aims to analyze the speaker's state and identify their emotion in the conversation. Recent works in ERC focus on context modeling but ignore the representation of contextual emotional tendency. In order to extract multi-modal information and the emotional tendency of the utterance effectively, we propose a new structure named Emoformer to extract multi-modal emotion vectors from different modalities and fuse them with sentence vector to be an emotion capsule. Furthermore, we design an end-to-end ERC model called EmoCaps, which extracts emotion vectors through the Emoformer structure and obtain the emotion classification results from a context analysis model. Through the experiments with two benchmark datasets, our model shows better performance than the existing state-of-the-art models.

show abstract

Section: So We Propose a New Multi-modal Emotionalmentioning

confidence: 99%

EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition

Li¹,

Tang²,

Zhao³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

show abstract

“…Our proposed method outperforms all three baselines, obtaining 3.54% and 3.25% absolute improvement on WA and UA respectively over the prior state of the art. Sentiment Analysis We adopt CMU-MOSEI (Zadeh et al 2018) dataset to evaluate the sentiment analysis task, which aims to predict the degree of positive and negative sentiment. We follow the same experimental protocol as MuIT (Tsai et al 2019), with the same train/test data split.…”

Section: Fine-tuning On Downstream Tasksmentioning

confidence: 99%

Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data

Kang¹,

Liu²,

Li³

et al. 2022

AAAI

View full text Add to dashboard Cite

Multimodal pre-training for audio-and-text has recently been proved to be effective and has significantly improved the performance of many downstream speech understanding tasks. However, these state-of-the-art pre-training audio-text models work well only when provided with large amount of parallel audio-and-text data, which brings challenges on many languages that are rich in unimodal corpora but scarce of parallel cross-modal corpus. In this paper, we investigate whether it is possible to pre-train an audio-text multimodal model with extremely low-resource parallel data and extra non-parallel unimodal data. Our pre-training framework consists of the following components: (1) Intra-modal Denoising Auto-Encoding (IDAE), which is able to reconstruct input text (audio) representations from a noisy version of itself. (2) Cross-modal Denoising Auto-Encoding (CDAE), which is pre-trained to reconstruct the input text (audio), given both a noisy version of the input text (audio) and the corresponding translated noisy audio features (text embeddings). (3) Iterative Denoising Process (IDP), which iteratively translates raw audio (text) and the corresponding text embeddings (audio features) translated from previous iteration into the new less-noisy text embeddings (audio features). We adapt a dual cross-modal Transformer as our backbone model which consists of two unimodal encoders for IDAE and two cross-modal encoders for CDAE and IDP. Our method achieves comparable performance on multiple downstream speech understanding tasks compared with the model pre-trained on fully parallel data, demonstrating the great potential of the proposed method.

show abstract

“…CMU-MOSI (Wöllmer et al, 2013) is sentiment prediction taks on a set of short youtube video clips. CMU-MOSEI (Zadeh et al, 2018b) is a similar dataset consisting of around 23k review videos taken from YouTube. The output in both cases is a sentiment score in [−3, 3].…”

Section: Datasetsmentioning

confidence: 99%

Multimodal fusion via cortical network inspired losses

Shankar¹

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Information integration from different modalities is an active area of research. Human beings and, in general, biological neural systems are quite adept at using a multitude of signals from different sensory perceptive fields to interact with the environment and each other. Recent work in deep fusion models via neural networks has led to substantial improvements over unimodal approaches in areas like speech recognition, emotion recognition and analysis, captioning and image description. However, such research has mostly focused on architectural changes allowing for fusion of different modalities while keeping the model complexity manageable. Inspired by neuroscientific ideas about multisensory integration and processing, we investigate the effect of introducing neural dependencies in the loss functions. Experiments on multimodal sentiment analysis tasks with different models show that our approach provides a consistent performance boost.

show abstract

Multi-attention Recurrent Network for Human Communication Comprehension

Cited by 326 publications

References 20 publications

EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition

EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition

Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data

Multimodal fusion via cortical network inspired losses

Contact Info

Product

Resources

About