Differential Attention for Visual Question Answering

Patro, Badri N.; Namboodiri, Vinay P.

doi:10.1109/cvpr.2018.00801

Cited by 70 publications

(53 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The current dominant framework for VQA systems consists of an image encoder, a question encoder, multimodal fusion, and an answer predictor. In lieu of directly using visual features from CNN-based feature extractors, [56,11,41,33,49,38,63,36] explored various image attention mechanisms to locate regions that are relevant to the question. To learn a better representation of the question, [33,38,11] proposed to perform question-guided image attention and image-guided question attention collaboratively, to merge knowledge from both visual and textual modalities in the encoding stage.…”

Section: Visual Question Answeringmentioning

confidence: 99%

Relation-Aware Graph Attention Network for Visual Question Answering

Gan

Cheng

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

344

201

View full text Add to dashboard Cite

In order to answer semantically-complicated questions about an image, a Visual Question Answering (VQA) model needs to fully understand the visual scene in the image, especially the interactive dynamics between different objects. We propose a Relation-aware Graph Attention Network (ReGAT), which encodes each image into a graph and models multi-type inter-object relations via a graph attention mechanism, to learn question-adaptive relation representations. Two types of visual object relations are explored: (i) Explicit Relations that represent geometric positions and semantic interactions between objects; and (ii) Implicit Relations that capture the hidden dynamics between image regions. Experiments demonstrate that ReGAT outperforms prior state-of-the-art approaches on both VQA 2.0 and VQA-CP v2 datasets. We further show that Re-GAT is compatible to existing VQA architectures, and can be used as a generic relation encoder to boost the model performance for VQA. 1

show abstract

Section: Visual Question Answeringmentioning

confidence: 99%

Relation-Aware Graph Attention Network for Visual Question Answering

Gan

Cheng

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

344

201

View full text Add to dashboard Cite

show abstract

“…The visual attention mechanism has been widely applied to form joint representations of input questions and images, which are subsequently handled by a classifier to produce answer predictions. Recent years have seen significant improvement in terms of performance, by either enhancing the visual attention module (Xu and Saenko, 2016 ; Yang et al, 2016 ; Kazemi and Elqursh, 2017 ; Anderson et al, 2018 ; Patro and Namboodiri, 2018 ), or improving quality of the joint embedding (Fukui et al, 2016 ; Lu et al, 2016 ; Noh et al, 2016 ; Ben-Younes et al, 2017 ; Yu et al, 2017 ). With model ensemble, the current state-of-the-art model has achieved over 72% accuracy (Jiang et al, 2018 ) on the VQA v2.0 test set.…”

Section: Related Workmentioning

confidence: 99%

“…There is little relationship modeling between the question modality and image modality, so that it looks more like a black box without interpretable process. Some recent works (Fukui et al, 2016 ; Lu et al, 2016 ; Noh et al, 2016 ; Xu and Saenko, 2016 ; Ben-Younes et al, 2017 ; Kazemi and Elqursh, 2017 ; Yu et al, 2017 ; Anderson et al, 2018 ; Kim et al, 2018 ; Patro and Namboodiri, 2018 ) introduce the attention mechanism into VQA models to attend questions to salient regions of input images, so that the joint embedding of attended regions and questions carries more accurate information for question answering. With model ensemble, attention based VQA models can achieve over 72% prediction accuracy (Jiang et al, 2018 ) on the test set of the VQA v2.0 dataset (Goyal et al, 2017 ).…”

Section: Introductionmentioning

confidence: 99%

Toward Accurate Visual Reasoning With Dual-Path Neural Module Networks

Li³

et al. 2020

Front. Robot. AI

View full text Add to dashboard Cite

Visual reasoning is a critical stage in visual question answering (Antol et al., 2015 ), but most of the state-of-the-art methods categorized the VQA tasks as a classification problem without taking the reasoning process into account. Various approaches are proposed to solve this multi-modal task that requires both abilities of comprehension and reasoning. The recently proposed neural module network (Andreas et al., 2016b ), which assembles the model with a few primitive modules, is capable of performing a spatial or arithmetical reasoning over the input image to answer the questions. Nevertheless, its performance is not satisfying especially in the real-world datasets (e.g., VQA 1.0& 2.0) due to its limited primitive modules and suboptimal layout. To address these issues, we propose a novel method of Dual-Path Neural Module Network which can implement complex visual reasoning by forming a more flexible layout regularized by the pairwise loss. Specifically, we first use the region proposal network to generate both visual and spatial information, which helps it perform spatial reasoning. Then, we advocate to process a pair of different images along with the same question simultaneously, named as a “complementary pair,” which encourages the model to learn a more reasonable layout by suppressing the overfitting to the language priors. The model can jointly learn the parameters in the primitive module and the layout generation policy, which is further boosted by introducing a novel pairwise reward. Extensive experiments show that our approach significantly improves the performance of neural module networks especially on the real-world datasets.

show abstract

“…Image QA (Antol et al 2015;Gao et al 2015;Ren, Kiros, and Zemel 2015b;Lu et al 2018b;Nam, Ha, and Kim 2017;Yang et al 2016;Xu and Saenko 2016;Lu et al 2016;Patro and Namboodiri 2018;Teney et al 2018), the task that infers answers to questions on a given image, has achieved much progress recently. Based on the framework of image captioning, most early works adopt typical CNN-RNN models, which use Convolutional Neural Networks (CNN) to extract image features and use Recurrent Neural Networks (RNN) to represent question information.…”

Section: Image Question Answeringmentioning

confidence: 99%

Structured Two-Stream Attention Network for Video Question Answering

Gao

Zeng

Song

et al. 2019

AAAI

View full text Add to dashboard Cite

To date, visual question answering (VQA) (i.e., image QA and video QA) is still a holy grail in vision and language understanding, especially for video QA. Compared with image QA that focuses primarily on understanding the associations between image region-level details and corresponding questions, video QA requires a model to jointly reason across both spatial and long-range temporal structures of a video as well as text to provide an accurate answer. In this paper, we specifically tackle the problem of video QA by proposing a Structured Two-stream Attention network, namely STA, to answer a free-form or open-ended natural language question about the content of a given video. First, we infer rich longrange temporal structures in videos using our structured segment component and encode text features. Then, our structured two-stream attention component simultaneously localizes important visual instance, reduces the influence of background video and focuses on the relevant text. Finally, the structured two-stream fusion component incorporates different segments of query and video aware context representation and infers the answers. Experiments on the large-scale video QA dataset TGIF-QA show that our proposed method significantly surpasses the best counterpart (i.e., with one representation for the video input) by 13.0%, 13.5%, 11.0% and 0.3 for Action, Trans., TrameQA and Count tasks. It also outperforms the best competitor (i.e., with two representations) on the Action, Trans., TrameQA tasks by 4.1%, 4.7%, and 5.1%.

show abstract

Differential Attention for Visual Question Answering

Cited by 70 publications

References 21 publications

Relation-Aware Graph Attention Network for Visual Question Answering

Relation-Aware Graph Attention Network for Visual Question Answering

Toward Accurate Visual Reasoning With Dual-Path Neural Module Networks

Structured Two-Stream Attention Network for Video Question Answering

Contact Info

Product

Resources

About