Detecting Human-Object Interaction via Fabricated Compositional Learning

Huang, Zhi; Yu, Baosheng; Qiao, Yu; Peng, Xiaojiang; Tao, Dacheng

doi:10.1109/cvpr46437.2021.01441

Cited by 76 publications

(33 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some works also exploit graph structure to enhance object dependencies [27,28,31,33,40]. Another bunch of two-stage methods is the compositional approaches [13][14][15]20], which disentangle HOI representations by learning from fabricated compositional HOIs. In contrast, our method disentangles representations by disentangled task encoders and decoders and its one-stage framework does not rely on pre-computed object proposals.…”

Section: Two-stage Methodsmentioning

confidence: 99%

“…To compose HOI triplets, it generates additional associative embeddings to match the interactions and instances. Since HOI detection is a composition problem [13,15], the latter decomposing strategy has several advantages compared with unified multi-tasking strategy. First, two sub-task decoders arXiv:2204.09290v1 [cs.CV] 20 Apr 2022 might attend to different regions via cross attention to facilitate learning and also results in better interpretability.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

End-to-End Human Object Interaction Detection with HOI Transformer

Cheng

Wang

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

194

111

View full text Add to dashboard Cite

Human-Object Interaction Detection tackles the problem of joint localization and classification of human object interactions. Existing HOI transformers either adopt a single decoder for triplet prediction, or utilize two parallel decoders to detect individual objects and interactions separately, and compose triplets by a matching process. In contrast, we decouple the triplet prediction into human-object pair detection and interaction classification. Our main motivation is that detecting the human-object instances and classifying interactions accurately needs to learn representations that focus on different regions. To this end, we present Disentangled Transformer, where both encoder and decoder are disentangled to facilitate learning of two subtasks. To associate the predictions of disentangled decoders, we first generate a unified representation for HOI triplets with a base decoder, and then utilize it as input feature of each disentangled decoder. Extensive experiments show that our method outperforms prior work on two public HOI benchmarks by a sizeable margin. Code will be available.

show abstract

Section: Two-stage Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

End-to-End Human Object Interaction Detection with HOI Transformer

Cheng

Wang

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

194

111

View full text Add to dashboard Cite

show abstract

“…For UC, compared with FCL [15], GEN-VLKT s achieves 38.85% and 22.41% relative mAP gains on full categories for rare first and non-rare first selections, respectively. Specifically, benefiting from the VLKT mechanism, GEN-VLKT s still significantly promotes the performance for the unseen categories without the feature factorization and composition among images like VCL [13], FCL [15] and ATL [14]. The improvements mainly come from the utilization of CLIP, as indicated by comparing GEN-VLKT s to the baseline.…”

Section: Effectiveness For Zero-shot Hoi Detectionmentioning

confidence: 99%

“…VCL [13] composed novel HOI samples by combining decomposed object and verb features with pair-wise images and within images. FCL [15] presented an object fabricator to generate fake object representations for rare and unseen HOIs. ATL [14] explored object affordances from additional object images to discover novel HOI categories.…”

Section: Interaction Scoresmentioning

confidence: 99%

GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection

Liao¹,

Zhang²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

The task of Human-Object Interaction (HOI) detection could be divided into two core problems, i.e., human-object association and interaction understanding. In this paper, we reveal and address the disadvantages of the conventional query-driven HOI detectors from the two aspects. For the association, previous two-branch methods suffer from complex and costly post-matching, while single-branch methods ignore the features distinction in different tasks. We propose Guided-Embedding Network (GEN) to attain a twobranch pipeline without post-matching. In GEN, we design an instance decoder to detect humans and objects with two independent query sets and a position Guided Embedding (p-GE) to mark the human and object in the same position as a pair. Besides, we design an interaction decoder to classify interactions, where the interaction queries are made of instance Guided Embeddings (i-GE) generated from the outputs of each instance decoder layer. For the interaction understanding, previous methods suffer from long-tailed distribution and zero-shot discovery. This paper proposes Visual-Linguistic Knowledge Transfer (VLKT) training strategy to enhance interaction understanding by transferring knowledge from a visual-linguistic pre-trained model CLIP. In specific, we extract text embeddings for all labels with CLIP to initialize the classifier and adopt a mimic loss to minimize the visual feature distance between GEN and CLIP. As a result, GEN-VLKT outperforms the state of the art by large margins on multiple datasets, e.g., +5.05 mAP on HICO-Det. The source codes are available at https://github.com/YueLiao/gen-vlkt.

show abstract

“…HOI, is of particular interest to understand the interactions of humans and other objects. A lot of HOI benchmarks, such as HICO [4], COCOa [39], vCOCO [10], and HOI-COCO [14], are built on top of the object categories provided in the COCO dataset [27].…”

Section: Related Workmentioning

confidence: 99%

Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions

Jiang

Nie³

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

A significant gap remains between today's visual pattern recognition models and human-level visual cognition especially when it comes to few-shot learning and compositional reasoning of novel concepts. We introduce Bongard-HOI, a new visual reasoning benchmark that focuses on compositional learning of human-object interactions (HOIs) from natural images. It is inspired by two desirable characteristics from the classical Bongard problems (BPs): 1) few-shot concept learning, and 2) context-dependent reasoning. We carefully curate the few-shot instances with hard negatives, where positive and negative images only disagree on action labels, making mere recognition of object categories insufficient to complete our benchmarks. We also design multiple test sets to systematically study the generalization of visual learning models, where we vary the overlap of the HOI concepts between the training and test sets of fewshot instances, from partial to no overlaps. Bongard-HOI presents a substantial challenge to today's visual recognition models. The state-of-the-art HOI detection model achieves only 62% accuracy on few-shot binary prediction while even amateur human testers on MTurk have 91% accuracy. With the Bongard-HOI benchmark, we hope to further advance research efforts in visual reasoning, especially in holistic perception-reasoning systems and better representation learning. Code is available. 1

show abstract

Detecting Human-Object Interaction via Fabricated Compositional Learning

Cited by 76 publications

References 52 publications

End-to-End Human Object Interaction Detection with HOI Transformer

End-to-End Human Object Interaction Detection with HOI Transformer

GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection

Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions

Contact Info

Product

Resources

About