Research on Visual Question Answering Based on GAT Relational Reasoning

Miao, Yongwu; Cheng, Wenfang; He, Shuyun; Jiang, Hui

doi:10.1007/s11063-021-10689-2

Cited by 11 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At present, AI is developing rapidly. As one of the applications of intelligent systems, VQA has attracted more Chinese researchers and scientists, focusing on this frontier field and promoting research progress in related fields ( Guo & Han, 2022 ; Guo & Han, 2023 ; Miao et al, 2022b ; Miao et al, 2022a ; Peng et al, 2022a ; Shen et al, 2022 ; Liu et al, 2022a ).…”

Section: Survey Methodologymentioning

confidence: 99%

The multi-modal fusion in visual question answering: a review of attention mechanisms

Liu

Yin

et al. 2023

PeerJ Computer Science

186

View full text Add to dashboard Cite

Visual Question Answering (VQA) is a significant cross-disciplinary issue in the fields of computer vision and natural language processing that requires a computer to output a natural language answer based on pictures and questions posed based on the pictures. This requires simultaneous processing of multimodal fusion of text features and visual features, and the key task that can ensure its success is the attention mechanism. Bringing in attention mechanisms makes it better to integrate text features and image features into a compact multi-modal representation. Therefore, it is necessary to clarify the development status of attention mechanism, understand the most advanced attention mechanism methods, and look forward to its future development direction. In this article, we first conduct a bibliometric analysis of the correlation through CiteSpace, then we find and reasonably speculate that the attention mechanism has great development potential in cross-modal retrieval. Secondly, we discuss the classification and application of existing attention mechanisms in VQA tasks, analysis their shortcomings, and summarize current improvement methods. Finally, through the continuous exploration of attention mechanisms, we believe that VQA will evolve in a smarter and more human direction.

show abstract

Section: Survey Methodologymentioning

confidence: 99%

The multi-modal fusion in visual question answering: a review of attention mechanisms

Liu

Yin

et al. 2023

PeerJ Computer Science

186

View full text Add to dashboard Cite

show abstract

“…Cao et al [45] proposed a Graph Matching Attention network that constructs a graph from the input image and question and infers the relationships between different modalities using cross-modality graph matching attention. Miao et al [46] organized an image as a scene graph and used question-guided node attention to construct a dynamic scene graph. Guo et al [47] proposed SCAVQAN, which sets thresholds for attention scores to filter out features that are not helpful for predicting the answer.…”

Section: Multimodal Interactionmentioning

confidence: 99%

Multimodal Joint Representation by Shared Knowledge Embedding with Vision-Language View for VQA

Cheng,

Jiang,

et al. 2024

Preprint

View full text Add to dashboard Cite

Current works in Visual Question Answering(VQA) that introduce external knowledge mainly focus on leveraging the knowledge to supplement the language representation in VQA model’s question input. However, such approaches ignore the commonsense information implied in the image. In this paper, we propose a novel VQA framework to embed knowledge features into vision and language respectively with a shared knowledge graph. To bridge the gap between visual representation and knowledge representation, we propose the knowledge enhancing visual representation (KEVR) module. KEVR is designed to explore external knowledge in image from the knowledge graph. By utilizing KEVR, external knowledge related to the objects in the image can be embedded into visual representation directly. In terms of the input question, a designed transformer is utilized to embed knowledge features into language representation. The knowledge graph used in our model is extracted from three knowledge bases. We organize the prior knowledge in the form of RDF triples to establish knowledge connections, then a graph neural network is employed to extract multilateral relationships in knowledge graph. What's more, a two-stream transformer is employed to get the attention based vision-language representation. The results of our experiments show that our model outperforms the best baseline model with an accuracy of 1.34% and 2.59% on VQA 2.0 and OK-VQA datasets respectively.

show abstract

“…To address the question that visual question and answer models neglect the modeling of object relationships in image, Cheng [42] proposed the relational reasoning mode Graph Attention Network Relation Reasoning(GAT2R) by introducing a graph attention mechanism. GAT2R model includes question feature extraction, scene graph generation, scene graph update part, multimodal fusion and answer prediction.…”

Section: B Attentional Mechanismsmentioning

confidence: 99%

Research on 3D Reconstruction of Furniture Based on Differentiable Renderer

Miao¹,

Jiang²,

Jiang³

et al. 2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

Due to the self-obscuration, traditional 3D reconstruction algorithms have difficulty in recovering the 3D structure of an object from a single image. With the rapid development of convolutional neural networks, 3D reconstruction based on deep learning has attracted a wide range of attention from researchers. However, it is expensive to obtain the 3D supervised data corresponding to the objects. To solve the above problems, we combine convolutional neural networks with differentiable renderer and propose the Mesh_CA in this paper, which enables reconstruction of a single image without 3D supervision data. Specifically, an ellipsoid is first initialized for each input single view, and then the features extracted by the convolutional neural network are used to guide the deformation of the ellipsoid to obtain the generated 3D object; After that, the generated object is passed into the differentiable renderer and its corresponding contour information is output; finally, calculating the error between the predicted contour and the real one, and the final 3D object is obtained after training and testing. By training and testing on five types of furniture objects on a large-scale public dataset ShapeNet, the performance of the proposed Mesh_CA surpasses current classical methods.

show abstract

Research on Visual Question Answering Based on GAT Relational Reasoning

Cited by 11 publications

References 27 publications

The multi-modal fusion in visual question answering: a review of attention mechanisms

The multi-modal fusion in visual question answering: a review of attention mechanisms

Multimodal Joint Representation by Shared Knowledge Embedding with Vision-Language View for VQA

Research on 3D Reconstruction of Furniture Based on Differentiable Renderer

Contact Info

Product

Resources

About