GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering

Liang, Weixin; Jiang, Yanhao; Liu, Zixuan

doi:10.48550/arxiv.2104.10283

Cited by 4 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…al [50] created a new benchmark, OK-VQA that contains most of the open domain questions where information provided in image and question alone is not enough to answer the question. Many authors have used a graph-based approach [17,41,42,43] to integrate external KB into the VQA model where the important objects in the visual image represent a node in a graph and the relationship between these nodes as edge. [38] proposed a novel methodology that takes the question's textual keywords and important visual items from the image, using these to extract knowledge from ConceptNet in the form of a knowledge graph.…”

Section: Knowledge Incorporated Vqa Modelsmentioning

confidence: 99%

A Comprehensive Review and Open Challenges on Visual Question Answering Models

Shaik,

Koshti,

Gupta

et al. 2023

AiBi Revista de Investigación, Administración e Ingeniería

View full text Add to dashboard Cite

Users are now able to actively interact with images and pose different questions based on images, thanks to recent developments in artificial intelligence. In turn, a response in a natural language answer is expected. The study discusses a variety of datasets that can be used to examine applications for visual question-answering (VQA), as well as their advantages and disadvantages. Four different forms of VQA models—simple joint embedding-based models, attention-based models, knowledge-incorporated models, and domain-specific VQA models—are in-depth examined in this article. We also critically assess the drawbacks and future possibilities of all current state-of-the-art (SoTa), end-to-end VQA models. Finally, we present the directions and guidelines for further development of the VQA models.

show abstract

Section: Knowledge Incorporated Vqa Modelsmentioning

confidence: 99%

A Comprehensive Review and Open Challenges on Visual Question Answering Models

Shaik,

Koshti,

Gupta

et al. 2023

AiBi Revista de Investigación, Administración e Ingeniería

View full text Add to dashboard Cite

show abstract

“…To answer VQA questions, models typically leverage variants of attention to obtain a representation of the image that is relevant to the question (Andreas et al, 2016;Yang et al, 2015;Xu and Saenko, 2016;Fukui et al, 2016;Lu et al, 2016). A plethora of works (Liang et al, 2021;Hudson and Manning, 2018;Yi et al, 2018b;Xiong et al, 2016;Kim et al, 2018;Teney et al, 2017a) have attempted to enhance the reasoning capability of VQA models, with Teney et al (2017a) proposing to improve VQA using structured representations of the scene contents and questions. They developed a deep neural network that leverages the structure in these representations and builds graphs over scene objects and question words.…”

Section: Multimodal Question Answeringmentioning

confidence: 99%

Multimodal Graph Transformer for Multimodal Question Answering

He,

Wang

2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

Despite the success of Transformer models in vision and language tasks, they often learn knowledge from enormous data implicitly and cannot utilize structured input data directly. On the other hand, structured learning approaches such as graph neural networks (GNNs) that integrate prior information can barely compete with Transformer models. In this work, we aim to benefit from both worlds and propose a novel Multimodal Graph Transformer for question answering tasks that requires performing reasoning across multiple modalities. We introduce a graph-involved plug-and-play quasi-attention mechanism to incorporate multimodal graph information, acquired from text and visual data, to the vanilla self-attention as effective prior. In particular, we construct the text graph, dense region graph, and semantic graph to generate adjacency matrices, and then compose them with input vision and language features to perform downstream reasoning. Such a way of regularizing self-attention with graph information significantly improves the inferring ability and helps align features from different modalities. We validate the effectiveness of Multimodal Graph Transformer over its Transformer baselines on GQA, VQAv2, and MultiModalQA datasets.

show abstract

“…Summaira, Jabeen, et al [17] conducted an extensive study on multimodal deep learning, incorporating modalities such as image, video, text, audio, body gestures, facial expressions, and physiological signals. Among those, there have been studies applying multimodal deep learning combining images and text, including visual-based referred expression understanding and phrase localization [18,19], as well as image and video captioning [20][21][22], textto-image generation [23][24][25], and visual question answering (VQA) [26][27][28][29][30]. Antol et al [26] developed a multimodal deep learning model for the visual question answering (VQA) task.…”

Section: Application Of Multimodal Techniques In Other Fieldsmentioning

confidence: 99%

A Multimodal Deep Learning Model Using Text, Image, and Code Data for Improving Issue Classification Tasks

Kwak,

Jung,

Lee

2023

Applied Sciences

View full text Add to dashboard Cite

Issue reports are valuable resources for the continuous maintenance and improvement of software. Managing issue reports requires a significant effort from developers. To address this problem, many researchers have proposed automated techniques for classifying issue reports. However, those techniques fall short of yielding reasonable classification accuracy. We notice that those techniques rely on text-based unimodal models. In this paper, we propose a novel multimodal model-based classification technique to use heterogeneous information in issue reports for issue classification. The proposed technique combines information from text, images, and code of issue reports. To evaluate the proposed technique, we conduct experiments with four different projects. The experiments compare the performance of the proposed technique with text-based unimodal models. Our experimental results show that the proposed technique achieves a 5.07% to 14.12% higher F1-score than the text-based unimodal models. Our findings demonstrate that utilizing heterogeneous data of issue reports helps improve the performance of issue classification.

show abstract

GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering

Cited by 4 publications

References 18 publications

A Comprehensive Review and Open Challenges on Visual Question Answering Models

A Comprehensive Review and Open Challenges on Visual Question Answering Models

Multimodal Graph Transformer for Multimodal Question Answering

A Multimodal Deep Learning Model Using Text, Image, and Code Data for Improving Issue Classification Tasks

Contact Info

Product

Resources

About