Fine-Grained Scene Graph Generation with Data Transfer

Abstract

Scene graph generation (SGG) aims to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. However, due to the data distribution problems including longtail distribution and semantic ambiguity, the predictions of current SGG models tend to collapse to several frequent but uninformative predicates (e.g., on, at), which limits practical application of these models in downstream tasks. To deal with the problems above, we propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a play-and-plug fashion and expanded to large SGG with 1,807 predicate classes. Our IETrans tries to relieve the data distribution problem by automatically creating an enhanced dataset that provides more sufficient and coherent annotations for all predicates. By training on the transferred dataset, a Neural Motif model doubles the macro performance while maintaining competitive micro performance. The data and code for this paper are publicly available at https://github.com/ waxnkw/IETrans-SGG.pytorch.

Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World

“…Since the predicates in scene graphs are highly relevant to the context, the direct enhancement methods based on class distribution are inapplicable for the balanced scene graph generation. Thus, some methods in SGG [48,52] explore visually relevant relationships from the external knowledge base to enhance the existing dataset. However, previous approaches require additional hyper-parameters or hand-designed enhancement rules limited to pre-defined scene graphs.…”

“…Although several weakly-supervised approaches could improve visual relation modeling through different knowledge bases [50,36,48,52], they require hand-designed rules and thus have limited generalization ability. Thus, we attempt to utilize the linguistic knowledge of pre-trained language models to boost fine-grained predicates in a lowresource way and make language models flexibly aware of scenes through visual prompts, as shown in the visuallyprompted language model module of CaCao in Figure 2 (a).…”

“…Furthermore, we explore more challenging datasets (i.e. GQA-200 [16,22], VG-1800 [52]) where predicates are more diverse to validate CaCao's generalization ability in large-scale scenarios. Data Split.…”

“…Evaluation and Metrics. Following recent works [52,31], we evaluate our model on three widely used SGG tasks: PredCls, SGCls, and SGDet. Since the Recall@K of all predicates could be easily affected by biased distribution, it cannot precisely evaluate models' performance on long-tail distribution SGG.…”

“…Another alternative way is to increase the number of tail predicates in training. IETrans [52] uses internal relation correlation to enhance the existing dataset. However, these methods rely on the prior distribution of source data and only work in specific pre-defined conditions.…”