Dual Self-Guided Attention with Sparse Question Networks for Visual Question Answering

Shen, Xiang; Han, Dong‐Guk; Chang, Chin‐Chen; Zong, Liang

doi:10.1587/transinf.2021edp7189

Cited by 7 publications

(6 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At present, AI is developing rapidly. As one of the applications of intelligent systems, VQA has attracted more Chinese researchers and scientists, focusing on this frontier field and promoting research progress in related fields ( Guo & Han, 2022 ; Guo & Han, 2023 ; Miao et al, 2022b ; Miao et al, 2022a ; Peng et al, 2022a ; Shen et al, 2022 ; Liu et al, 2022a ).…”

Section: Survey Methodologymentioning

confidence: 99%

The multi-modal fusion in visual question answering: a review of attention mechanisms

Liu

Yin

et al. 2023

PeerJ Computer Science

184

View full text Add to dashboard Cite

Visual Question Answering (VQA) is a significant cross-disciplinary issue in the fields of computer vision and natural language processing that requires a computer to output a natural language answer based on pictures and questions posed based on the pictures. This requires simultaneous processing of multimodal fusion of text features and visual features, and the key task that can ensure its success is the attention mechanism. Bringing in attention mechanisms makes it better to integrate text features and image features into a compact multi-modal representation. Therefore, it is necessary to clarify the development status of attention mechanism, understand the most advanced attention mechanism methods, and look forward to its future development direction. In this article, we first conduct a bibliometric analysis of the correlation through CiteSpace, then we find and reasonably speculate that the attention mechanism has great development potential in cross-modal retrieval. Secondly, we discuss the classification and application of existing attention mechanisms in VQA tasks, analysis their shortcomings, and summarize current improvement methods. Finally, through the continuous exploration of attention mechanisms, we believe that VQA will evolve in a smarter and more human direction.

show abstract

Section: Survey Methodologymentioning

confidence: 99%

The multi-modal fusion in visual question answering: a review of attention mechanisms

Liu

Yin

et al. 2023

PeerJ Computer Science

184

View full text Add to dashboard Cite

show abstract

“…While these models achieve great results, they can significantly improve the model’s performance by pre-training the base model and transferring it to downstream tasks based on large-scale visual and question datasets. However, in the research of network models for VQA tasks and downstream tasks, utilizing the end-to-end approach [ 7 – 9 , 12 , 16 , 17 , 20 ] to train network models can better capture the modal information of images and texts and can effectively improve model performance.…”

Section: Related Workmentioning

confidence: 99%

“…The image representation can guide the attention of the question, and the performance of the question can guide the attention of the image. Nam et al [ 9 ] proposed a dual attention network to collect necessary information through multi-step processing of specific areas in the image and keywords in the question. However, because these attention models learn multimodal coarse interaction examples, it is difficult to infer the correlation images and questions between them.…”

Section: Related Workmentioning

confidence: 99%

“…For example, [7-10, 24, 25] use CNN-based networks to explore various image attention mechanisms to localize question-related regions. At the same time, there are also related works [8,9,26] that propose methods of using question-guided image attention and image-guided question attention and modal fusion of the extracted image and question features through the decoder and encoder. Multimodal fusion is crucial in VQA models.…”

Section: Visual Question Answeringmentioning

confidence: 99%

“…In recent years, researchers have explored the multi-modal learning and visual reasoning of text and image features. The most advanced VQA method [7][8][9][10][11] is mainly focused on learning the multi-modal joint representation of images and questions. Specifically, the early proposed visual question answering model uses Convolutional Neural Network (CNN) to extract global features of an image and a Bag-of-words (BOW) model to extract text features of the question.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An effective spatial relational reasoning networks for visual question answering

et al. 2022

Self Cite

View full text Add to dashboard Cite

Visual Question Answering (VQA) is a method of answering questions in natural language based on the content of images and has been widely concerned by researchers. The existing research on the visual question answering model mainly focuses on the point of view of attention mechanism and multi-modal fusion. It only pays attention to the visual semantic features of the image in the process of image modeling, ignoring the importance of modeling the spatial relationship of visual objects. We are aiming at the existing problems of the existing VQA model research. An effective spatial relationship reasoning network model is proposed, which can combine visual object semantic reasoning and spatial relationship reasoning at the same time to realize fine-grained multi-modal reasoning and fusion. A sparse attention encoder is designed to capture contextual information effectively in the semantic reasoning module. In the spatial relationship reasoning module, the graph neural network attention mechanism is used to model the spatial relationship of visual objects, which can correctly answer complex spatial relationship reasoning questions. Finally, a practical compact self-attention (CSA) mechanism is designed to reduce the redundancy of self-attention in linear transformation and the number of model parameters and effectively improve the model’s overall performance. Quantitative and qualitative experiments are conducted on the benchmark datasets of VQA 2.0 and GQA. The experimental results demonstrate that the proposed method performs favorably against the state-of-the-art approaches. Our best single model has an overall accuracy of 71.18% on the VQA 2.0 dataset and 57.59% on the GQA dataset.

show abstract

Modular dual-stream visual fusion network for visual question answering

Xue,

Wang,

Wang

et al. 2024

Vis Comput

View full text Add to dashboard Cite

Dual Self-Guided Attention with Sparse Question Networks for Visual Question Answering

Cited by 7 publications

References 41 publications

The multi-modal fusion in visual question answering: a review of attention mechanisms

The multi-modal fusion in visual question answering: a review of attention mechanisms

An effective spatial relational reasoning networks for visual question answering

Modular dual-stream visual fusion network for visual question answering

Contact Info

Product

Resources

About