MUREL: Multimodal Relational Reasoning for Visual Question Answering

Abstract

Multimodal attentional networks are currently state-ofthe-art models for Visual Question Answering (VQA) tasks involving real images. Although attention allows to focus on the visual content relevant to the question, this simple mechanism is arguably insufficient to model complex reasoning features required for VQA or other high-level tasks.In this paper, we propose MuRel, a multimodal relational network which is learned end-to-end to reason over real images. Our first contribution is the introduction of the MuRel cell, an atomic reasoning primitive representing interactions between question and image regions by a rich vectorial representation, and modeling region relations with pairwise combinations. Secondly, we incorporate the cell into a full MuRel network, which progressively refines visual and question interactions, and can be leveraged to define visualization schemes finer than mere attention maps.We validate the relevance of our approach with various ablation studies, and show its superiority to attentionbased methods on three datasets: VQA 2.0, VQA-CP v2 and TDIUC. Our final MuRel network is competitive to or outperforms state-of-the-art results in this challenging context.Our code is available: github.com/Cadene/ murel.bootstrap.pytorch * Equal contribution DonutStep #1Step #2 Step #3 MuRel network ✔

Relation-Aware Graph Attention Network for Visual Question Answering

“…Another line of research focuses on implicit relations, where no explicit semantic or spatial relations are used to construct the graph. Instead, all the relations are implicitly captured by an attention module or via higher-order methods over the fully-connected graph of an input image [46,21,6,57], to model the interactions between detected objects. For example, [46] reasons over all the possible pairs of objects in an image via the use of simple MLPs.…”

Activity-driven Weakly-Supervised Spatio-Temporal Grounding from Untrimmed Videos

“…Fully-supervised Spatio-Temporal Grounding has been developed by combining with other tasks, such as object tracking [32], video captioning [34] and visual question answering [2]. Yang et al [32] add language description on an object tracking dataset [10] to make it for grounding task and propose a grounding and tracking integration model.…”

“…"they". Video grounding on untrimmed videos is of great significance while more challenging than trimmed video, since the untrimmed video contains large temporal incoherence caused by camera motion and camera shot cut 2 .…”

K-armed Bandit based Multi-Modal Network Architecture Search for Visual Question Answering

“…Typically, VQA is often considered as a classification task with fixed categories [2,4,10,57]. In this case, to obtain the optimal joint representations for question answering, some VQA methods focus on exploring the full interactions between two modalities [6,10,22,23,[60][61][62]. Early developments resorted to bilinear-based approaches to improve the one-layer fusion of two modalities, e.g., compact bilinear pooling [10], low-rank bilinear pooling [23] and factorized bilinear pooling [61].…”