Scene Graph Generation With Hierarchical Context

Ren, Guina; Ren, Lejian; Liao, Yue; Liu, Si; Li, Bo; Han, Jizhong; Yan, Shuicheng

doi:10.1109/tnnls.2020.2979270

Cited by 36 publications

(17 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recall@50 Recall@100 MSDN [19] 42.3 48.2 Message passing [20] 44.8 53.0 CISC [33] 53.2 57.9 Graph-RCNN [21] 54.2 59.1 Neural-Motifs [22] 65.2 67.1 KERN [30] 65.8 67.8 HCNet [34] 66.4 68.8 SGRN [23] 66.5 68.7 L-SVRU [24] 68. We include different variants of our model named S2G-V, S2G-V+D and S2G-V+S to study the role of visual, depth and spatial information in images.…”

Section: Methodsmentioning

confidence: 99%

“…Finally, the scene graph is generated based on these two graph neural networks. Moreover, Rowan et al [22], Sanghyun et al [31], Wang et al [33] and Ren et al [34] utilize global context to explore relations between objects for building scene graphs where as, Yang et al [21] and [36] capture sub-graph structures in images.…”

Section: Related Workmentioning

confidence: 99%

“…R@50 and R@100 are abbreviations of Recall@50 and Recall@100. The baselines include text-based [19], graphbased [21], [22], context-aware models [33], [34] and message passing models [20], [23], [30] originally designed for visual relationship detection.…”

Section: A Quantitative Analysismentioning

confidence: 99%

See 2 more Smart Citations

Scene Graph Generation Using Depth, Spatial, and Visual Cues in 2D Images

Kumar¹,

Nair

2022

IEEE Access

View full text Add to dashboard Cite

To understand an image or a scene properly, it is necessary to identify objects participating in the scene, their relationships, and various attributes that describe their properties. A scene graph is a high-level representation that confines all these features in a structured manner. Scene graph generation includes multiple challenges like the semantics of relationships considered and the availability of a wellbalanced dataset with sufficient training examples. We tried to mitigate these problems by extracting two subsets, VG-R10 and VG-A16, from the popular Visual Genome dataset. Also, a framework (S2G) is proposed for generating scene graphs directly from images using depth and spatial information of object pairs. Evaluations on the scene graph generation model reveal that the proposed framework achieves better results on our data than the state-of-the-art.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Scene Graph Generation Using Depth, Spatial, and Visual Cues in 2D Images

Kumar¹,

Nair

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…SGG is to generate a visuallygrounded scene graph that most accurately correlates with an image. Most of the early methods (Lu et al, 2016;Yu et al, 2017;Dai et al, 2017;Yang et al, 2018b;Li et al, 2018;Mi and Chen, 2020;Xu et al, 2017;Ren et al, 2020;Hung et al, 2020;Guo et al, 2021) focused on better modeling the interactions among objects. Xu et al (Xu et al, 2017) firstly proposed a SGG task, and introduced a message passing mechanism to improve the relationship representation.…”

Section: Related Workmentioning

confidence: 99%

Predicate correlation learning for scene graph generation

Tao,

Mi,

et al. 2021

Preprint

View full text Add to dashboard Cite

For a typical Scene Graph Generation (SGG) method, there is often a large gap in the performance of the predicates' head classes and tail classes. This phenomenon is mainly caused by the semantic overlap between different predicates as well as the long-tailed data distribution. In this paper, a Predicate Correlation Learning (PCL) method for SGG is proposed to address the above two problems by taking the correlation between predicates into consideration. To describe the semantic overlap between strong-correlated predicate classes, a Predicate Correlation Matrix (PCM) is defined to quantify the relationship between predicate pairs, which is dynamically updated to remove the matrix's long-tailed bias. In addition, PCM is integrated into a Predicate Correlation Loss function (L PC ) to reduce discouraging gradients of unannotated classes. The proposed method is evaluated on Visual Genome benchmark, where the performance of the tail classes is significantly improved when built on the existing methods.

show abstract

“…Early works usually use human pose to recognize action [20,45]. Some works use additional contextual cue [7,13,18,40,41] or a bayesian model [13] to solve this task. Recently some deep-based methods achieve promising results on HOI recognition [11,31,34].…”

Section: Related Work 21 Human-object Interactions Detectionmentioning

confidence: 99%

InteractGAN: Learning to Generate Human-Object Interaction

Gao

Liu

Zhu

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

Compared with the widely studied Human-Object Interaction DETection (HOI-DET), no effort has been devoted to its inverse problem, i.e. to generate an HOI scene image according to the given relationship triplet , to our best knowledge. We term this new task "Human-Object Interaction Image Generation" (HOI-IG). HOI-IG is a research-worthy task with great application prospects, such as online shopping, film production and interactive entertainment. In this work, we introduce an Interact-GAN to solve this challenging task. Our method is composed of two stages: (1) manipulating the posture of a given human image conditioned on a predicate. (2) merging the transformed human image and object image to one realistic scene image while satisfying their expected relative position and ratio. Besides, to address the large spatial misalignment issue caused by fusing two images content with reasonable spatial layout, we propose a Relation-based Spatial Transformer Network (RSTN) to adaptively process the images conditioned on their interaction. Extensive experiments on two challenging datasets demonstrate the effectiveness and superiority of our approach. We advocate for the image generation community to draw more attention to the new Human-Object Interaction Image Generation problem. To facilitate future research, our project will be released at: http://colalab.org/projects/InteractGAN.

show abstract

Scene Graph Generation With Hierarchical Context

Cited by 36 publications

References 26 publications

Scene Graph Generation Using Depth, Spatial, and Visual Cues in 2D Images

Scene Graph Generation Using Depth, Spatial, and Visual Cues in 2D Images

Predicate correlation learning for scene graph generation

InteractGAN: Learning to Generate Human-Object Interaction

Contact Info

Product

Resources

About