Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation

Goel, Arushi; Fernando, Basura; Keller, Frank; Bilen, Hakan

doi:10.1109/cvpr52688.2022.01515

Cited by 20 publications

(6 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The GSI method is compared with three baseline models, Motifs, VCTree, and the Transformer, as well as their corresponding improved methods: TDE, STL [31], PCPL, CogTree, NARE [32], CAME [33] and LS-KD. In addition, the mR@K values of the IMP [34], KERN [35], GPS-Net [36], BGNN and NLS [37] models are listed in Table 2.…”

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

Geometric and Semantic Improvement for Unbiased Scene Graph Generation

2023

KSII TIIS

View full text Add to dashboard Cite

Scene graphs are structured representations that can clearly convey objects and the relationships between them, but are often heavily biased due to the highly skewed, long-tailed relational labeling in the dataset. Indeed, the visual world itself and its descriptions are biased. Therefore, Unbiased Scene Graph Generation (USGG) prefers to train models to eliminate long-tail effects as much as possible, rather than altering the dataset directly. To this end, we propose Geometric and Semantic Improvement (GSI) for USGG to mitigate this issue. First, to fully exploit the feature information in the images, geometric dimension and semantic dimension enhancement modules are designed. The geometric module is designed from the perspective that the position information between neighboring object pairs will affect each other, which can improve the recall rate of the overall relationship in the dataset. The semantic module further processes the embedded word vector, which can enhance the acquisition of semantic information. Then, to improve the recall rate of the tail data, the Class Balanced Seesaw Loss (CBSLoss) is designed for the tail data. The recall rate of the prediction is improved by penalizing the body or tail relations that are judged incorrectly in the dataset. The experimental findings demonstrate that the GSI method performs better than mainstream models in terms of the mean Recall@K (mR@K) metric in three tasks. The long-tailed imbalance in the Visual Genome 150 (VG150) dataset is addressed better using the GSI method than by most of the existing methods.

show abstract

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

Geometric and Semantic Improvement for Unbiased Scene Graph Generation

2023

KSII TIIS

View full text Add to dashboard Cite

show abstract

“…Zero-shot Scene Graph Generation. Current zero-shot SGG mainly focuses on the generalization of relation combination, ignoring to predict of novel predicates in the wild world [30,39,20,12]. He et al…”

Section: Related Workmentioning

confidence: 99%

Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World

Yu¹,

Li²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Scene Graph Generation (SGG) aims to extract relationships in images for vision understanding. Although recent works have made steady progress on SGG, they still suffer long-tail distribution issues that tail-predicates are more costly to train and hard to distinguish due to a small amount of annotated data compared to frequent predicates. Existing re-balancing strategies try to haddle it via prior rules but are still confined to pre-defined conditions, which are not scalable for various models and datasets. In this paper, we propose a Crossmodal prediCate boosting (CaCao) framework, where a visually-prompted language model is learned to generate diverse fine-grained predicates in a low-resource way. The proposed CaCao can be applied in a plug-and-play fashion and automatically strengthen existing SGG to tackle the long-tailed problem. Based on that, we further introduce a novel Entangled cross-modal prompt approach for open-world predicate scene graph generation (Epic), where models can generalize to unseen predicates in a zeroshot manner. Comprehensive experiments on three benchmark datasets show that CaCao consistently boosts the performance of multiple scene graph generation models in a model-agnostic way. Moreover, our Epic achieves competitive performance on open-world predicate prediction.

show abstract

“…Furthermore, to alleviate the imbalanced relationship distribution, Yin et al [38] reformulated the conventional one-hot classification as a n-hot multiclass hierarchical recognition via novel Intra-Hierarchical trees (IH-trees) for each label set in the triplet subject, predicate, object . Recently, unbiased SGG [43,[47][48][49][50][51][52][53][54][62][63][64][65][66][67][68][69][70][71] has drawn unprecedented interest for more generalized SGG models. Occurrence-based Node Priority Sensitive (NPS)-loss [47] was used for balancing predictions; the Total Direct Effect (TDE) method has proposed firstly for unbiased learning by Tang et al [43], which directly separates the bias from biased predictions through the counterfactual methodologies on causal graphs; CogTree [48] addressed the debiasing issue based on the coarse-to-fine structure of the relationships from the cognition viewpoint; Li et al [49] improved the context modeling for tail categories by designing the bipartite graph network and message propagation on resampled objects and images.…”

Section: Related Workmentioning

confidence: 99%

Skew Class-Balanced Re-Weighting for Unbiased Scene Graph Generation

Kang

Yoo

2023

MAKE

View full text Add to dashboard Cite

An unbiased scene graph generation (SGG) algorithm referred to as Skew Class-Balanced Re-Weighting (SCR) is proposed for considering the unbiased predicate prediction caused by the long-tailed distribution. The prior works focus mainly on alleviating the deteriorating performances of the minority predicate predictions, showing drastic dropping recall scores, i.e., losing the majority predicate performances. It has not yet correctly analyzed the trade-off between majority and minority predicate performances in the limited SGG datasets. In this paper, to alleviate the issue, the Skew Class-Balanced Re-Weighting (SCR) loss function is considered for the unbiased SGG models. Leveraged by the skewness of biased predicate predictions, the SCR estimates the target predicate weight coefficient and then re-weights more to the biased predicates for better trading-off between the majority predicates and the minority ones. Extensive experiments conducted on the standard Visual Genome dataset and Open Image V4 and V6 show the performances and generality of the SCR with the traditional SGG models.

show abstract

Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation

Cited by 20 publications

References 44 publications

Geometric and Semantic Improvement for Unbiased Scene Graph Generation

Geometric and Semantic Improvement for Unbiased Scene Graph Generation

Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World

Skew Class-Balanced Re-Weighting for Unbiased Scene Graph Generation

Contact Info

Product

Resources

About