Joint Learning of Object Graph and Relation Graph for Visual Question Answering

Li, Hao; Li, Xu; Karimi, Belhal; Chen, Jie; Sun, Mingming

doi:10.1109/icme52920.2022.9859766

Cited by 13 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Exploring further, Graphhopper [56] tackles the complexity of multi-hop reasoning over complex visual scenes to deduce reasoning paths leading to answers in VQA. The Dual Message-passing enhanced GNN (DM-GNN) [63] encodes multi-scale scene graph information into two distinct graphs focusing on objects and relations. This dual structure achieves a balanced representation of object, relation, and attribute features in VQA.…”

Section: Visual Question Answeringmentioning

confidence: 99%

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

Khan,

Ilievski,

Breslin

et al. 2024

NAI

View full text Add to dashboard Cite

Combining deep learning and common sense knowledge via neurosymbolic integration is essential for semantically rich scene representation and intuitive visual reasoning. This survey paper delves into data- and knowledge-driven scene representation and visual reasoning approaches based on deep learning, common sense knowledge and neurosymbolic integration. It explores how scene graph generation, a process that detects and analyses objects, visual relationships and attributes in scenes, serves as a symbolic scene representation. This representation forms the basis for higher-level visual reasoning tasks such as visual question answering, image captioning, image retrieval, image generation, and multimodal event processing. Infusing common sense knowledge, particularly through the use of heterogeneous knowledge graphs, improves the accuracy, expressiveness and reasoning ability of the representation and allows for intuitive downstream reasoning. Neurosymbolic integration in these approaches ranges from loose to tight coupling of neural and symbolic components. The paper reviews and categorises the state-of-the-art knowledge-based neurosymbolic approaches for scene representation based on the types of deep learning architecture, common sense knowledge source and neurosymbolic integration used. The paper also discusses the visual reasoning tasks, datasets, evaluation metrics, key challenges and future directions, providing a comprehensive review of this research area and motivating further research into knowledge-enhanced and data-driven neurosymbolic scene representation and visual reasoning.

show abstract

Section: Visual Question Answeringmentioning

confidence: 99%

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

Khan,

Ilievski,

Breslin

et al. 2024

NAI

View full text Add to dashboard Cite

show abstract

“…ReGAT [11] further introduces the semantic position edge relations into the spatial graph, which trains a classifier to predict the spatial relationships between visual objects into 11 specific categories, for example, ‘cover’, ‘inter‐sect’ and ‘no‐relation’ for too far pairs. DM‐GNN [12] simultaneously introduces the relative position and human behaviour information from the image by utilising the SGG model to construct the visual relationship scene graph. KG‐Aug [36] uses the entities of visual objects and key question words to retrieve the corresponding triplets in the ConceptNet, and combines these triplets to construct the knowledge scene graph.…”

Section: Related Workmentioning

confidence: 99%

“…Here, the spatial scene graph reliably assists in solving the position information problems, but it fails to introduce the relevant semantic relationships between objects. In addition, DM‐GNN [12] utilises the existing Scene Graph Generation [10] (SGG) model to construct a visual relationship based scene graph for one image, which includes the position relationships and semantic relationships about human behaviour. Here, SGG is pre‐trained on the large visual relationship benchmark (e.g.…”

Section: Introductionmentioning

confidence: 99%

DSGEM: Dual scene graph enhancement module‐based visual question answering

Wang

et al. 2023

IET Computer Vision

View full text Add to dashboard Cite

Visual Question Answering (VQA) aims to appropriately answer a text question by understanding the image content. Attention‐based VQA models mine the implicit relationships between objects according to the feature similarity, which neglects the explicit relationships between objects, for example, the relative position. Most Visual Scene Graph‐based VQA models exploit the relative positions or visual relationships between objects to construct the visual scene graph, while they suffer from the semantic insufficiency of visual edge relations. Besides, the scene graph of text modality is often ignored in these works. In this article, a novel Dual Scene Graph Enhancement Module (DSGEM) is proposed that exploits the relevant external knowledge to simultaneously construct two interpretable scene graph structures of image and text modalities, which makes the reasoning process more logical and precise. Specifically, the authors respectively build the visual and textual scene graphs with the help of commonsense knowledge and syntactic structure, which explicitly endows the specific semantics to each edge relation. Then, two scene graph enhancement modules are proposed to propagate the involved external and structural knowledge to explicitly guide the feature interaction between objects (nodes). Finally, the authors embed such two scene graph enhancement modules to existing VQA models to introduce the explicit relation reasoning ability. Experimental results on both VQA V2 and OK‐VQA datasets show that the proposed DSGEM is effective and compatible to various VQA architectures.

show abstract

“…Graph representation learning intends to transform nodes and links on the graph into lower-dimensional vector embeddings, which can be quite challenging due to the complex graph topological structures and node/link attributes. While approaches on static graphs have made breakthroughs and demonstrated distinguishable applicability in various fields (Graepel et al 2010;He et al 2014;Li et al 2022;Zhu et al 2022), those on temporal graphs are just getting started. Modeling a temporal graph (which may evolve over time with the addition, deletion, and changing of its attributes) is a core problem in developing real-world industrial systems (e.g., social network, citation network, recommendation systems) where many data are time-dependent, and is Figure 1: An example of temporal graph modeling.…”

Section: Introductionmentioning

confidence: 99%

FTM: A Frame-Level Timeline Modeling Method for Temporal Graph Representation Learning

Cao

et al. 2023

AAAI

View full text Add to dashboard Cite

Learning representations for graph-structured data is essential for graph analytical tasks. While remarkable progress has been made on static graphs, researches on temporal graphs are still in its beginning stage. The bottleneck of the temporal graph representation learning approach is the neighborhood aggregation strategy, based on which graph attributes share and gather information explicitly. Existing neighborhood aggregation strategies fail to capture either the short-term features or the long-term features of temporal graph attributes, leading to unsatisfactory model performance and even poor robustness and domain generality of the representation learning method. To address this problem, we propose a Frame-level Timeline Modeling (FTM) method that helps to capture both short-term and long-term features and thus learns more informative representations on temporal graphs. In particular, we present a novel link-based framing technique to preserve the short-term features and then incorporate a timeline aggregator module to capture the intrinsic dynamics of graph evolution as long-term features. Our method can be easily assembled with most temporal GNNs. Extensive experiments on common datasets show that our method brings great improvements to the capability, robustness, and domain generality of backbone methods in downstream tasks. Our code can be found at https://github.com/yeeeqichen/FTM.

show abstract

Joint Learning of Object Graph and Relation Graph for Visual Question Answering

Cited by 13 publications

References 6 publications

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

A survey of neurosymbolic visual reasoning with scene graphs and common sense knowledge

DSGEM: Dual scene graph enhancement module‐based visual question answering

FTM: A Frame-Level Timeline Modeling Method for Temporal Graph Representation Learning

Contact Info

Product

Resources

About