DRAKE: Deep Pair-Wise Relation Alignment for Knowledge-Enhanced Multimodal Scene Graph Generation in Social Media Posts

Fu, Ze; Zheng, Changmeng; Feng, Junhao; Cai, Yi; Wei, Xiaoyong; Wang, Yaowei; Li, Qing

doi:10.1109/tcsvt.2022.3231437

Cited by 5 publications

(1 citation statement)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…H UMAN-OBJECT interaction (HOI) detection is a process of detecting interaction between a human and an object in an image [1]. Precisely estimating human-object interactions can greatly improve various visual understanding tasks, such as image retrieval [2], visual question answering [3], and scene graph generation [4], [5]. Given a series of ⟨"Human", "Object", "Verb"⟩ triples, an HOI detector is needed to locate humanobject pairs and identify their interactions.…”

Section: Introductionmentioning

confidence: 99%

Zero-shot Object Detection Through Vision-Language Embedding Alignment

Xie

Zheng²

2022

2022 IEEE International Conference on Data Mining Workshops (ICDMW)

View full text Add to dashboard Cite

Human-object interaction (HOI) detection aims to locate human-object pairs and identify their interaction categories in images. Most existing methods primarily focus on supervised learning, which relies on extensive manual HOI annotations. In this paper, we propose a novel framework, termed Knowledge Integration to HOI (KI2HOI), that effectively integrates the knowledge of visual-language model to improve zero-shot HOI detection. Specifically, the verb feature learning module is designed based on visual semantics, by employing the verb extraction decoder to convert corresponding verb queries into interaction-specific category representations. We develop an effective additive self-attention mechanism to generate more comprehensive visual representations. Moreover, the innovative interaction representation decoder effectively extracts informative regions by integrating spatial and visual feature information through a cross-attention mechanism. To deal with zero-shot learning in low-data, we leverage a priori knowledge from the CLIP text encoder to initialize the linear classifier for enhanced interaction understanding. Extensive experiments conducted on the mainstream HICO-DET and V-COCO datasets demonstrate that our model outperforms the previous methods in various zero-shot and full-supervised settings.

show abstract