MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Yang, Jaeyoung; Wang, Yongxin; Yi, Ruitao; Zhu, Yezi; Rehman, Azaan; Zadeh, Amir; Poria, Soujanya; Morency, Louis–Philippe

doi:10.48550/arxiv.2010.11985

Cited by 5 publications

(11 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, certain multimodal language approaches freeze a Transformer and aim to fuse all modalities using tensor outer-products [23,24], Canonical Correlation Analysis based methods [25], Attentive LSTM based methods [26,27,8,28,19], sequence to sequence based methods [29], cross-modal Transformer-based methods [6,30,31], graph-based method [32], and multi-task learning [7].…”

Section: Human Multimodal Language Analysismentioning

confidence: 99%

TEASEL: A Transformer-Based Speech-Prefixed Language Model

Arjmand¹,

Dousti²,

Moradi³

2021

Preprint

View full text Add to dashboard Cite

Multimodal language analysis is a burgeoning field of NLP that aims to simultaneously model a speaker's words, acoustical annotations, and facial expressions. In this area, lexicon features usually outperform other modalities because they are pre-trained on large corpora via Transformer-based models. Despite their strong performance, training a new self-supervised learning (SSL) Transformer on any modality is not usually attainable due to insufficient data, which is the case in multimodal language learning. This work proposes a Transformer-Based Speech-Prefixed Language Model called TEASEL to approach the mentioned constraints without training a complete Transformer model. TEASEL model includes speech modality as a dynamic prefix besides the textual modality compared to a conventional language model. This method exploits a conventional pre-trained language model as a cross-modal Transformer model. We evaluated TEASEL for the multimodal sentiment analysis task defined by CMU-MOSI dataset. Extensive experiments show that our model outperforms unimodal baseline language models by 4% and outperforms the current multimodal state-of-the-art (SoTA) model by 1% in F1-score. Additionally, our proposed method is 72% smaller than the SoTA model.

show abstract

Section: Human Multimodal Language Analysismentioning

confidence: 99%

TEASEL: A Transformer-Based Speech-Prefixed Language Model

Arjmand¹,

Dousti²,

Moradi³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…With RNN being the main modules, they are confronted with the problems of training and long inferring time. Recently, [16,24,30] propose alternative networks to model unaligned multimodal sequences. Tsai et al [24] use cross-modal transformer and self-attention transformer to learn long-range dependency.…”

Section: Related Work 21 Human Multimodal Language Analysismentioning

confidence: 99%

“…In contrast, our proposed GraphCAGE replaces the self-attention transformer with graph-based model which produces more refined and high-level representations of sequences. In [16] and [30], sequences are transformed into graphs and GCN is applied to learn long-range dependency, which not only avoid the problems of RNN but also successfully model unaligned multimodal sequences. Nevertheless, they implement graph pooling and edge pruning to drop some nodes in order to obtain the final representation of graph, leading to information loss.…”

Section: Related Work 21 Human Multimodal Language Analysismentioning

confidence: 99%

“…Similarly, in order to obtain high-level graph representation, edge pruning [30] is usually applied in each GCN layer. However, pooling and pruning may rudely drop some important nodes, leading to the loss of information.…”

Section: Graph Neural Networkmentioning

confidence: 99%

“…For example, in [16] and [30], the authors define node and edge based on multimodal sequences processed only by Feed-Forward-Network, which causes that the created graph is not highly expressive because the node embedding is not built on high-level features. Moreover, they regard every time step as a node and only depend on GCN to learn long-range dependency, leading to insufficient learning.…”

Section: Node Definitionmentioning

confidence: 99%

See 2 more Smart Citations

Graph Capsule Aggregation for Unaligned Multimodal Sequences

Mai

2021

Proceedings of the 2021 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Humans express their opinions and emotions through multiple modalities which mainly consist of textual, acoustic and visual modalities. Prior works on multimodal sentiment analysis mostly apply Recurrent Neural Network (RNN) to model aligned multimodal sequences. However, it is unpractical to align multimodal sequences due to different sample rates for different modalities. Moreover, RNN is prone to the issues of gradient vanishing or exploding and it has limited capacity of learning long-range dependency which is the major obstacle to model unaligned multimodal sequences. In this paper, we introduce Graph Capsule Aggregation (GraphCAGE) to model unaligned multimodal sequences with graph-based neural model and Capsule Network. By converting sequence data into graph, the previously mentioned problems of RNN are avoided. In addition, the aggregation capability of Capsule Network and the graph-based structure enable our model to be interpretable and better solve the problem of longrange dependency. Experimental results suggest that GraphCAGE achieves state-of-the-art performance on two benchmark datasets with representations refined by Capsule Network and interpretation provided. CCS CONCEPTS• Information systems → Multimedia streaming.

show abstract

Toward Explainable Affective Computing: A Review

Cortiñas-Lorenzo,

Lacey

2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Affective computing has an unprecedented potential to change the way humans interact with technology. While the last decades have witnessed vast progress in the field, multimodal affective computing systems are generally black box by design. As affective systems start to be deployed in real-world scenarios, such as education or healthcare, a shift of focus toward improved transparency and interpretability is needed. In this context, how do we explain the output of affective computing models? and how to do so without limiting predictive performance? In this article, we review affective computing work from an explainable AI (XAI) perspective, collecting and synthesizing relevant papers into three major XAI approaches: premodel (applied before training), in-model (applied during training), and postmodel (applied after training). We present and discuss the most fundamental challenges in the field, namely, how to relate explanations back to multimodal and time-dependent data, how to integrate context and inductive biases into explanations using mechanisms such as attention, generative modeling, or graph-based methods, and how to capture intramodal and cross-modal interactions in post hoc explanations. While explainable affective computing is still nascent, existing methods are promising, contributing not only toward improved transparency but, in many cases, surpassing state-of-the-art results. Based on these findings, we explore directions for future research and discuss the importance of data-driven XAI and explanation goals, and explainee needs definition, as well as causability or the extent to which a given method leads to human understanding.

show abstract

MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Cited by 5 publications

References 31 publications

TEASEL: A Transformer-Based Speech-Prefixed Language Model

TEASEL: A Transformer-Based Speech-Prefixed Language Model

Graph Capsule Aggregation for Unaligned Multimodal Sequences

Toward Explainable Affective Computing: A Review

Contact Info

Product

Resources

About