Reformulating HOI Detection as Adaptive Set Prediction

Ming-fei, Chen; Liao, Yue; Liu, Si; Chen, Zhiyuan; Wang, Fei; Qian, Chen

doi:10.1109/cvpr46437.2021.00889

Cited by 128 publications

(76 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing HOI detection methods can be roughly categorized into crop-based and crop-free fashions [18,19,20,21].…”

Section: Related Workmentioning

confidence: 99%

Effective Actor-centric Human-object Interaction Detection

Xu,

Li,

Zhang

et al. 2022

Preprint

View full text Add to dashboard Cite

While Human-Object Interaction(HOI) Detection has achieved tremendous advances in recent, it still remains challenging due to complex interactions with multiple humans and objects occurring in images, which would inevitably lead to ambiguities. Most existing methods either generate all human-object pair candidates and infer their relationships by cropped local features successively in a two-stage manner, or directly predict interaction points in a one-stage procedure. However, the lack of spatial configurations or reasoning steps of two-or one-stage methods respectively limits their performance in such complex scenes. To avoid this ambiguity, we propose a novel actor-centric framework. The main ideas are that when inferring interactions: 1) the non-local features of the entire image guided by actor position are obtained to model the relationship between the actor and context, and then 2) we use an object branch to generate pixel-wise interaction area prediction, where the interaction area denotes the object central area. Moreover, we also use an actor branch to get interaction prediction of the actor and propose a novel composition strategy based on center-point indexing to generate the final HOI prediction. Thanks to the usage of the non-local features and the partly-coupled property of the human-objects composition strategy, our proposed framework can detect HOI more accurately especially for complex images. Extensive experimental results show that our method achieves the state-of-the-art on the challenging V-COCO and HICO-DET benchmarks and is more robust especially in multiple persons and/or objects scenes.

show abstract

“…Existing HOI detection methods can be roughly categorized into crop-based and crop-free fashions [18,19,20,21].…”

Section: Related Workmentioning

confidence: 99%

Effective Actor-centric Human-object Interaction Detection

Xu,

Li,

Zhang

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…One-stage methods can be categorized into three types: i) point-based methods (Liao et al 2020;Zhong et al 2021) which infers heuristically-defined interaction points, ii) anchor-based methods (Kim et al 2020a) which detects union boxes, iii) DETR-based (Carion et al 2020) methods. Thanks to DETR and its ability to extract contextual cues (Vaswani et al 2017), several customized end-to-end models (Kim et al 2021;Zou et al 2021;Chen et al 2021;Tamura, Ohashi, and Yoshinaga 2021) originated from DETR have achieved promising results. However, there has not been an end-to-end method to incorporate semantics or explicitly utilize object-guided hierarchical relation priors.…”

Section: Related Workmentioning

confidence: 99%

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics

Hangjie¹,

Wang²,

Ni³

et al. 2022

Preprint

View full text Add to dashboard Cite

Human-Object Interaction (HOI) detection is an essential task to understand human-centric images from a fine-grained perspective. Although end-to-end HOI detection models thrive, their paradigm of parallel human/object detection and verb class prediction loses two-stage methods' merit: objectguided hierarchy. The object in one HOI triplet gives direct clues to the verb to be predicted. In this paper, we aim to boost end-to-end models with object-guided statistical priors. Specifically, We propose to utilize a Verb Semantic Model (VSM) and use semantic aggregation to profit from this object-guided hierarchy. Similarity KL (SKL) loss is proposed to optimize VSM to align with the HOI dataset's priors. To overcome the static semantic embedding problem, we propose to generate cross-modality-aware visual and semantic features by Cross-Modal Calibration (CMC). The above modules combined composes Object-guided Cross-modal Calibration Network (OCN). Experiments conducted on two popular HOI detection benchmarks demonstrate the significance of incorporating the statistical prior knowledge and produce state-of-the-art performances. More detailed analysis indicates proposed modules serve as a stronger verb predictor and a more superior method of utilizing prior knowledge. The codes are available at https://github.com/JacobYuan7/OCN-HOI-Benchmark.

show abstract

“…One-stage methods [20][21][22] execute object detection and HOI detection concurrently and pair them afterwards. Recent studies [23][24][25][26] achieve end-to-end HOI detection with a DETR [27] style network and benefit from the wider perception field of transformers [26].…”

Section: Related Workmentioning

confidence: 99%

Is Object Detection Necessary for Human-Object Interaction Recognition?

Jin,

Chen,

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper revisits human-object interaction (HOI) recognition at image level without using supervisions of object location and human pose. We name it detectionfree HOI recognition, in contrast to the existing detection-supervised approaches which rely on object and keypoint detections to achieve state of the art. With our method, not only the detection supervision is evitable, but superior performance can be achieved by properly using image-text pre-training (such as CLIP) and the proposed Log-Sum-Exp Sign (LSE-Sign) loss function. Specifically, using text embeddings of class labels to initialize the linear classifier is essential for leveraging the CLIP pre-trained image encoder. In addition, LSE-Sign loss facilitates learning from multiple labels on an imbalanced dataset by normalizing gradients over all classes in a softmax format. Surprisingly, our detection-free solution achieves 60.5 mAP on the HICO dataset, outperforming the detection-supervised state of the art by 13.4 mAP.

show abstract

Reformulating HOI Detection as Adaptive Set Prediction

Cited by 128 publications

References 31 publications

Effective Actor-centric Human-object Interaction Detection

Effective Actor-centric Human-object Interaction Detection

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics

Is Object Detection Necessary for Human-Object Interaction Recognition?

Contact Info

Product

Resources

About