Energy-Based Learning for Scene Graph Generation

Suhail, Mohammed; Mittal, Abhay; Siddiquie, Behjat; Broaddus, Chris; Eledath, Jayan; Medioni, Gérard; Sigal, Leonid

doi:10.1109/cvpr46437.2021.01372

Cited by 127 publications

(48 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They refers to methods that can be applied in a plug-and-play fashion. For this part, we include TDE [22], CogTree [32], EBM [21], DeC [9], and DLFE [5]. (2) Specific models.…”

Section: Models Predicate Classificationmentioning

confidence: 99%

Fine-Grained Scene Graph Generation with Data Transfer

Zhang¹,

Yuan²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Scene graph generation (SGG) aims to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. However, due to the data distribution problems including longtail distribution and semantic ambiguity, the predictions of current SGG models tend to collapse to several frequent but uninformative predicates (e.g., on, at), which limits practical application of these models in downstream tasks. To deal with the problems above, we propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a play-and-plug fashion and expanded to large SGG with 1,807 predicate classes. Our IETrans tries to relieve the data distribution problem by automatically creating an enhanced dataset that provides more sufficient and coherent annotations for all predicates. By training on the transferred dataset, a Neural Motif model doubles the macro performance while maintaining competitive micro performance. The data and code for this paper are publicly available at https://github.com/ waxnkw/IETrans-SGG.pytorch.

show abstract

“…They refers to methods that can be applied in a plug-and-play fashion. For this part, we include TDE [22], CogTree [32], EBM [21], DeC [9], and DLFE [5]. (2) Specific models.…”

Section: Models Predicate Classificationmentioning

confidence: 99%

Fine-Grained Scene Graph Generation with Data Transfer

Zhang¹,

Yuan²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Semantic knowledge can be utilized as an additional feature to infer scene graphs [2], [9], [11], [54], [55]. Furthermore, statistic priors and knowledge graphs have been introduced in [11], [56], [57], [58], [59].…”

Section: Scene Graph Generationmentioning

confidence: 99%

RelTR: Relation Transformer for Scene Graph Generation

Yang¹,

Yang²,

Rosenhahn³

2022

Preprint

View full text Add to dashboard Cite

Different objects in the same scene are more or less related to each other, but only a limited number of these relationships are noteworthy. Inspired by DETR, which excels in object detection, we view scene graph generation as a set prediction problem and propose an end-to-end scene graph generation model RelTR which has an encoder-decoder architecture. The encoder reasons about the visual feature context while the decoder infers a fixed-size set of triplets subject-predicate-object using different types of attention mechanisms with coupled subject and object queries. We design a set prediction loss performing the matching between the ground truth and predicted triplets for the end-to-end training. In contrast to most existing scene graph generation methods, RelTR is a one-stage method that predicts a set of relationships directly only using visual appearance without combining entities and labeling all possible predicates. Extensive experiments on the Visual Genome and Open Images V6 datasets demonstrate the superior performance and fast inference of our model.

show abstract

“…Based on causal inference, [47] used Total Direct Effect for unbiased SGG. [43] proposed an energy-based constraint to learn predicates in small numbers. [59] formulated the predicate tree structure and used tree-based classbalance loss for training.…”

Section: Scene Graph Generationmentioning

confidence: 99%

“…However, they ignore the dataset and task properties, limiting the performance. In order to deal with the long-tail effect and various biases [47] within the dataset, recent works including [7,17,43,47] move towards designing the learning framework (i.e., the optimization strategy) to improve the overall performance of several classic SGG models. Even with this progress, making the fine-grained predicate prediction is still challenging.…”

Section: Introductionmentioning

confidence: 99%

Hierarchical Memory Learning for Fine-Grained Scene Graph Generation

Deng¹,

Li²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

As far as Scene Graph Generation (SGG), coarse and fine predicates mix in the dataset due to the crowd-sourced labeling, and the long-tail problem is also pronounced. Given this tricky situation, many existing SGG methods treat the predicates equally and learn the model under the supervision of mixed-granularity predicates in one stage, leading to relatively coarse predictions. In order to alleviate the negative impact of the suboptimum mixedgranularity annotation and long-tail effect problems, this paper proposes a novel Hierarchical Memory Learning (HML) framework to learn the model from simple to complex, which is similar to the human beings' hierarchical memory learning process. After the autonomous partition of coarse and fine predicates, the model is first trained on the coarse predicates and then learns the fine predicates. In order to realize this hierarchical learning pattern, this paper, for the first time, formulates the HML framework using the new Concept Reconstruction (CR) and Model Reconstruction (MR) constraints. It is worth noticing that the HML framework can be taken as one general optimization strategy to improve various SGG models, and significant improvement can be achieved on the SGG benchmark (i.e., Visual Genome).

show abstract

Energy-Based Learning for Scene Graph Generation

Cited by 127 publications

References 12 publications

Fine-Grained Scene Graph Generation with Data Transfer

Fine-Grained Scene Graph Generation with Data Transfer

RelTR: Relation Transformer for Scene Graph Generation

Hierarchical Memory Learning for Fine-Grained Scene Graph Generation

Contact Info

Product

Resources

About