DIRV: Dense Interaction Region Voting for End-to-End Human-Object Interaction Detection

Abstract

Recent years, human-object interaction (HOI) detection has achieved impressive advances. However, conventional two-stage methods are usually slow in inference. On the other hand, existing one-stage methods mainly focus on the union regions of interactions, which introduce unnecessary visual information as disturbances to HOI detection. To tackle the problems above, we propose a novel one-stage HOI detection approach DIRV in this paper, based on a new concept called interaction region for the HOI problem. Unlike previous methods, our approach concentrates on the densely sampled interaction regions across different scales for each human-object pair, so as to capture the subtle visual features that is most essential to the interaction. Moreover, in order to compensate for the detection flaws of a single interaction region, we introduce a novel voting strategy that makes full use of those overlapped interaction regions in place of conventional Non-Maximal Suppression (NMS). Extensive experiments on two popular benchmarks: V-COCO and HICO-DET show that our approach outperforms existing state-of-the-arts by a large margin with the highest inference speed and lightest network architecture. Our code is publicly available at www.github.com/MVIG-SJTU/DIRV.

Research laboratory on the mechanics of smart materials and structures, Zhejiang University

“…Though effective, they suffer from expensive computation due to the sequential inference architecture [17], and are highly dependent on prior detection results. In contrast, the one-stage methods [58][59][60][61] jointly detect human-object pairs and classify the interactions in an end-to-end manner by associating humans and objects with predefined anchors, which can be union boxes [58,61] or interaction points [59,60]. Despite featuring fast inference, they heavily rely on hand-crafted post-processing to associate interactions with object detection results [10].…”

Detecting human-object interactions for human activity analysis

“…Parallel one-stage framework. [67][68][69][70] often apply a parallel structure to generate box candidates and predict interaction points. In the end, a matching module will aggregate the results from parallel branches and form the final HOI tuples.…”

“…(1) One-stage framework: inspired by existing methods [67][68][69][70][73][74][75][76][77], we aim to use one unified framework to accomplish box detection and interaction recognition. (2) Detection with external knowledge: we aim to leverage human pose knowledge [19,60,61,63,85] and natural language knowledge [21,25,81,82] to address the HOI detection problem.…”

“…The standard paradigm is to decompose the task into two parts: (1) human and object instances detection; (2) instance matching and interaction classification. Depending on whether the two parts are sequentially or in parallel, existing works can be further categorized as two-stage solutions [53,55,56,58,63] or end-to-end one-stage solutions [67][68][69]166]. Regardless of which type of solutions, existing methods usually instantiate the object detection part as generic detectors, e.g., Faster RCNN [8], CenterNet [111], RetineNet [10], etc.…”

“…Besides, we compose a dataset from 100DOH [165] and annotate them with 300 object categories for hand-object interaction. Existing HOI methods often decompose the detection into two stages [51,55,56,58] or parallel branches [67][68][69]166]. One stage (or branch) uses generic object detectors (e.g., Faster RCNN [8], CenterNet [67], etc.)…”