Predicate Correlation Learning for Scene Graph Generation

Tao, Leitian; Li, Mi; Li, Nannan; Cheng, Xianhang; Hu, Yaosi; Chen, Zhenzhong

doi:10.1109/tip.2022.3181511

Cited by 15 publications

(4 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Scene graph generation was first pioneered in [13] for image retrieval, and the task quickly gained further traction, as seen in e.g. [21,33,39,42,43]. Recently, a number of papers have identified the long-tailed distribution in image scene graphs and focused on generating unbiased scene graphs [8,9,[15][16][17]41].…”

Section: Related Workmentioning

confidence: 99%

Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation

Chen

Mettes³

et al. 2023

Proceedings of the 2023 ACM International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

This paper investigates the problem of scene graph generation in videos with the aim of capturing semantic relations between subjects and objects in the form of ⟨subject, predicate, object⟩ triplets. Recognizing the predicate between subject and object pairs is imbalanced and multi-label in nature, ranging from ubiquitous interactions such as spatial relationships (e.g. in front of ) to rare interactions such as twisting. In widely-used benchmarks such as Action Genome and VidOR, the imbalance ratio between the most and least frequent predicates reaches 3,218 and 3,408, respectively, surpassing even benchmarks specifically designed for long-tailed recognition. Due to the long-tailed distributions and label co-occurrences, recent state-of-the-art methods predominantly focus on the most frequently occurring predicate classes, ignoring those in the long tail. In this paper, we analyze the limitations of current approaches for scene graph generation in videos and identify a one-to-one correspondence between predicate frequency and recall performance. To make the step towards unbiased scene graph generation in videos, we introduce a multi-label meta-learning framework to deal with the biased predicate distribution. Our meta-learning framework learns a meta-weight network for each training sample over all possible label losses. We evaluate our approach on the Action Genome and VidOR benchmarks by building upon two current state-of-theart methods for each benchmark. The experiments demonstrate that the multi-label meta-weight network improves the performance for predicates in the long tail without compromising performance for head classes, resulting in better overall performance and favorable generalizability. Code: https://github.com/shanshuo/ML-MWN. CCS CONCEPTS• Computing methodologies → Scene understanding; Activity recognition and understanding.

show abstract

Section: Related Workmentioning

confidence: 99%

Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation

Chen

Mettes³

et al. 2023

Proceedings of the 2023 ACM International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

show abstract

“…To evaluate our proposed method on VG150, our approach combines four classical SGG models to represent predicate features, namely Motifs [29], VCTree [30], Transformer [49] and PENet [31], as shown in Table II. The results of the state-of-the-art methods that are being compared are divided into various debiased methods [20], [22], [26]- [28], [35], [57], [58], [60], [61] on classical models and specific SGG models [21], [23], [24], [27], [32], [56], [59].…”

Section: Comparison With State Of the Arts A) Vg150mentioning

confidence: 99%

Learning to Generate an Unbiased Scene Graph by Using Attribute-Guided Predicate Features

Wang

Yuan

Chen

2023

AAAI

View full text Add to dashboard Cite

Scene Graph Generation (SGG) aims to capture the semantic information in an image and build a structured representation, which facilitates downstream tasks. The current challenge in SGG is to tackle the biased predictions caused by the long-tailed distribution of predicates. Since multiple predicates in SGG are coupled in an image, existing data re-balancing methods cannot completely balance the head and tail predicates. In this work, a decoupled learning framework is proposed for unbiased scene graph generation by using attribute-guided predicate features to construct a balanced training set. Specifically, the predicate recognition is decoupled into Predicate Feature Representation Learning (PFRL) and predicate classifier training with a class-balanced predicate feature set, which is constructed by our proposed Attribute-guided Predicate Feature Generation (A-PFG) model. In the A-PFG model, we first define the class labels of and corresponding visual feature as attributes to describe a predicate. Then the predicate feature and the attribute embedding are mapped into a shared hidden space by a dual Variational Auto-encoder (VAE), and finally the synthetic predicate features are forced to learn the contextual information in the attributes via cross reconstruction and distribution alignment. To demonstrate the effectiveness of our proposed method, our decoupled learning framework and A-PFG model are applied to various SGG models. The empirical results show that our method is substantially improved on all benchmarks and achieves new state-of-the-art performance for unbiased scene graph generation. Our code is available at https://github.com/wanglei0618/A-PFG.

show abstract

“…Although these methods greatly improved context representations, the generated scene graphs are far from satisfactory due to the biased data distribution. To tackle the biased problem, some cleverly designed loss functions [15], [16], [20]- [22], [33]- [35] and re-sampling strategies [14], [17], [18], [36] have been introduced to generate unbiased scene graphs. [33] proposes a flexible reweighting method that utilizes the correlation among predicate classes to adaptively seek out appropriate loss weights.…”

Section: Related Work a Scene Graph Generationmentioning

confidence: 99%

“…To mitigate the above issue, various de-biasing SGG methods [14]- [20] have been proposed. The mainstream methods can be roughly divided into two types: 1) re-sampling, which cuts down the head samples or repeats the tail samples to balance the distribution of training data, e.g., GCL [18], BA-SGG [17].…”

mentioning

confidence: 99%

Dual-branch Hybrid Learning Network for Unbiased Scene Graph Generation

Zheng¹,

Gao²,

Lyu³

et al. 2022

Preprint

View full text Add to dashboard Cite

The current studies of Scene Graph Generation (SGG) focus on solving the long-tailed problem for generating unbiased scene graphs. However, most de-biasing methods overemphasize the tail predicates and underestimate head ones throughout training, thereby wrecking the representation ability of head predicate features. Furthermore, these impaired features from head predicates harm the learning of tail predicates. In fact, the inference of tail predicates heavily depends on the general patterns learned from head ones, e.g., "standing on" depends on "on". Thus, these de-biasing SGG methods can neither achieve excellent performance on tail predicates nor satisfying behaviors on head ones. To address this issue, we propose a Dual-branch Hybrid Learning network (DHL) to take care of both head predicates and tail ones for SGG, including a Coarsegrained Learning Branch (CLB) and a Fine-grained Learning Branch (FLB). Specifically, the CLB is responsible for learning expertise and robust features of head predicates, while the FLB is expected to predict informative tail predicates. Furthermore, DHL is equipped with a Branch Curriculum Schedule (BCS) to make the two branches work well together. Experiments show that our approach achieves a new state-of-the-art performance on VG and GQA datasets and makes a trade-off between the performance of tail predicates and head ones. Moreover, extensive experiments on two downstream tasks (i.e., Image Captioning and Sentence-to-Graph Retrieval) further verify the generalization and practicability of our method. Our code is available at https://github.com/aa200647963/SGG-DHL/.

show abstract

Predicate Correlation Learning for Scene Graph Generation

Cited by 15 publications

References 45 publications

Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation

Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation

Learning to Generate an Unbiased Scene Graph by Using Attribute-Guided Predicate Features

Dual-branch Hybrid Learning Network for Unbiased Scene Graph Generation

Contact Info

Product

Resources

About