Scene Graph Refinement Network for Visual Question Answering

Qian, Tianwen; Chen, Jingjing; Chen, Shaoxiang; Wu, Bo; Jiang, Yu-Gang

doi:10.1109/tmm.2022.3169065

Cited by 30 publications

(10 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Video Encoder. We uniformly sample T frames from V and extract their CNN (e.g., I3D (Qian et al 2023)) features. These features are contextually encoded using a video encoder ϕ v to yield frame features ϕ v (V ) = {v 1 , v 2 , .…”

Section: Pseudo-supervised Setupmentioning

confidence: 99%

“…Natural Language Video Localization (NLVL) is a fundamental multimodal understanding task that aims to align textual queries with relevant video segments. NLVL is a core component for various applications such as video moment retrieval (Cao et al 2022), video question answering (Qian et al 2023;Lei et al 2020a), and video editing (Gao et al 2022). Prior works have primarily explored supervised (Zeng et al 2020;Wang, Ma, and Jiang 2020;Soldan et al 2021;Liu et al 2021;Yu et al 2020; or weakly supervised (Mun, Cho, and Han 2020;Zhang et al 2020Zhang et al , 2021) NLVL methodologies, relying on annotated video-query data to various extents.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Commonsense for Zero-Shot Natural Language Video Localization

Holla,

Lourentzou

2024

AAAI

View full text Add to dashboard Cite

Zero-shot Natural Language-Video Localization (NLVL) methods have exhibited promising results in training NLVL models exclusively with raw video data by dynamically generating video segments and pseudo-query annotations. However, existing pseudo-queries often lack grounding in the source video, resulting in unstructured and disjointed content. In this paper, we investigate the effectiveness of commonsense reasoning in zero-shot NLVL. Specifically, we present CORONET, a zero-shot NLVL framework that leverages commonsense to bridge the gap between videos and generated pseudo-queries via a commonsense enhancement module. CORONET employs Graph Convolution Networks (GCN) to encode commonsense information extracted from a knowledge graph, conditioned on the video, and cross-attention mechanisms to enhance the encoded video and pseudo-query representations prior to localization. Through empirical evaluations on two benchmark datasets, we demonstrate that CORONET surpasses both zero-shot and weakly supervised baselines, achieving improvements up to 32.13% across various recall thresholds and up to 6.33% in mIoU. These results underscore the significance of leveraging commonsense reasoning for zero-shot NLVL.

show abstract

Section: Pseudo-supervised Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Commonsense for Zero-Shot Natural Language Video Localization

Holla,

Lourentzou

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Liu et al [ 15 ] used the RGB-D information of images to represent inaccurate depth features for extracting semantic information. Qian et al [ 6 ] propose to refine the scene graphs for improving the effectiveness and present a scene graph refinement network (SGR), which introduces a transformer-based refinement network to enhance the object and relation features for better classification. Wu et al [ 21 ] propose to enhance video captioning with deep-level object relationships that are adaptively explored during training and present a transitive visual relationship detection (TVRD) module.…”

Section: Related Workmentioning

confidence: 99%

“…Visual relationships are usually expressed as triples <subject–predicate–object> [ 3 , 4 , 5 ]. They play an essential role in higher-level vision tasks, such as visual question answering [ 6 ], image captioning [ 7 ], and image generation [ 8 ]. There are many promising results in visual relationship detection works.…”

Section: Introductionmentioning

confidence: 99%

Visual Relationship Detection with Multimodal Fusion and Reasoning

Xiao

2022

Sensors

View full text Add to dashboard Cite

Visual relationship detection aims to completely understand visual scenes and has recently received increasing attention. However, current methods only use the visual features of images to train the semantic network, which does not match human habits in which we know obvious features of scenes and infer covert states using common sense. Therefore, these methods cannot predict some hidden relationships of object-pairs from complex scenes. To address this problem, we propose unifying vision–language fusion and knowledge graph reasoning to combine visual feature embedding with external common sense knowledge to determine the visual relationships of objects. In addition, before training the relationship detection network, we devise an object–pair proposal module to solve the combination explosion problem. Extensive experiments show that our proposed method outperforms the state-of-the-art methods on the Visual Genome and Visual Relationship Detection datasets.

show abstract

“…The ultimate goal of scene graph generation is to produce such representations from raw images (or videos [14]). Such representations have proved valuable for a variety of higher-level AI tasks, e.g., image/video captioning [24,31], and visual question answering [34]. Scene graph generation models usually contain two message passing networks: one for object detection, and the other for relation prediction among the detected objects.…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Relation Alignment for Scene Graph Generation

Xu¹,

Liao²,

Sigal³

2023

Preprint

View full text Add to dashboard Cite

The goal of scene graph generation is to predict a graph from an input image, where nodes correspond to identified and localized objects and edges to their corresponding interaction predicates. Existing methods are trained in a fully supervised manner and focus on message passing mechanisms, loss functions, and/or bias mitigation. In this work we introduce a simple-yet-effective self-supervised relational alignment regularization designed to improve the scene graph generation performance. The proposed alignment is general and can be combined with any existing scene graph generation framework, where it is trained alongside the original model's objective. The alignment is achieved through distillation, where an auxiliary relation prediction branch, that mirrors and shares parameters with the supervised counterpart, is designed. In the auxiliary branch, relational input features are partially masked prior to message passing and predicate prediction. The predictions for masked relations are then aligned with the supervised counterparts after the message passing. We illustrate the effectiveness of this self-supervised relational alignment in conjunction with two scene graph generation architectures, SGTR [22] and Neural Motifs [51], and show that in both cases we achieve significantly improved performance.

show abstract

Scene Graph Refinement Network for Visual Question Answering

Cited by 30 publications

References 65 publications

Commonsense for Zero-Shot Natural Language Video Localization

Commonsense for Zero-Shot Natural Language Video Localization

Visual Relationship Detection with Multimodal Fusion and Reasoning

Self-Supervised Relation Alignment for Scene Graph Generation

Contact Info

Product

Resources

About