Multimodal Graph Transformer for Multimodal Question Answering

Abstract

Despite the success of Transformer models in vision and language tasks, they often learn knowledge from enormous data implicitly and cannot utilize structured input data directly. On the other hand, structured learning approaches such as graph neural networks (GNNs) that integrate prior information can barely compete with Transformer models. In this work, we aim to benefit from both worlds and propose a novel Multimodal Graph Transformer for question answering tasks that requires performing reasoning across multiple modalities. We introduce a graph-involved plug-and-play quasi-attention mechanism to incorporate multimodal graph information, acquired from text and visual data, to the vanilla self-attention as effective prior. In particular, we construct the text graph, dense region graph, and semantic graph to generate adjacency matrices, and then compose them with input vision and language features to perform downstream reasoning. Such a way of regularizing self-attention with graph information significantly improves the inferring ability and helps align features from different modalities. We validate the effectiveness of Multimodal Graph Transformer over its Transformer baselines on GQA, VQAv2, and MultiModalQA datasets.

Unifying Text, Tables, and Images for Multimodal Question Answering

“…Early studies (Hannan et al, 2020;Talmor et al, 2021) decompose MMQA into three single-modal QA models. To align different modalities, some latest studies employ graph structures (Yang et al, 2022a;He and Wang, 2023) to enhance the cross-modal interaction.…”

“…Furthermore, we employ prefix-tuning as our fine-tuning strategy, which adds a task-specific prefix (e.g., MMQA) to the input sequence as a prompt. (Yang et al, 2022a), MGT (He and Wang, 2023), ManymodalQA (Hannan et al, 2020), ORConvQA (Qu et al, 2020), MAE , Solar . Detailed descriptions of each baseline are provided in Appendix A.…”

“…• MGT (He and Wang, 2023) also employs graph structure to model interactions between different modalities. MGT utilizes Transformer as the backbone and combines multimodal graph learning from unstructured data with Transformers.…”

“…Preliminary attempts in MMQA (Hannan et al, 2020;Talmor et al, 2021) typically utilize separate QA systems for different knowledge modal- ities, which failed to bridge the challenging gap between multimodal sources. To integrate information across modalities, various graph structures (Yang et al, 2022a;He and Wang, 2023) are introduced into MMQA systems to build semantic connections between different modalities. However, the graph structure is typically less compatible and practical with language models, and can barely compete with the powerful pre-trained language models (PLMs).…”