ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection

Abstract

We consider the problem of Human-Object Interaction (HOI) Detection, which aims to locate and recognize HOI instances in the form of ⟨ℎ , , ⟩ in images. Most existing works treat HOIs as individual interaction categories, thus can not handle the problem of long-tail distribution and polysemy of action labels. We argue that multi-level consistencies among objects, actions and interactions are strong cues for generating semantic representations of rare or previously unseen HOIs. Leveraging the compositional and relational peculiarities of HOI labels, we propose ConsNet, a knowledge-aware framework that explicitly encodes the relations among objects, actions and interactions into an undirected graph called consistency graph, and exploits Graph Attention Networks (GATs) to propagate knowledge among HOI categories as well as their constituents. Our model takes visual features of candidate human-object pairs and word embeddings of HOI labels as inputs, maps them into visual-semantic joint embedding space and obtains detection results by measuring their similarities. We extensively evaluate our model on the challenging V-COCO and HICO-DET datasets, and results validate that our approach outperforms stateof-the-arts under both fully-supervised and zero-shot settings. CCS CONCEPTS • Computing methodologies → Activity recognition and understanding; Scene understanding.

Detecting Human-to-Human-or-Object (H2O) Interactions with DIABOLO

“…On the one hand, most of the previous methods adopt a two-stage strategy [10], [8], [3], [16], [29], [26], [27], [20], [24]. During the first stage, they use an external object detector as [9] to point out interacting candidates, and then, during the second stage, another network is dedicated to estimate interactions between the proposals.…”

“…More recently, improvement in the second stage have been proposed by using additional information in the image. For instance, [16], [27], [20] use human pose to have a finer analysis of the posture of the interaction subject. Other methods add word embedding [24], [21] or segmentation [29].…”

GTNet:Guided Transformer Network for Detecting Human-Object Interactions

“…Human-Object Interaction: With the introduction of benchmark datasets like V-COCO [18] and HICO-DET [6], there is a plethora of works detecting human-object interactions [11,14,30,33,47,51,13,44,38,57,22,34,20,49,28]. Earlier works [17] in this area focus on the visual features of humans and objects.…”

“…Recently, graph-based architectures where humans and objects are considered nodes attempt to understand spatial context [13,49] for the structural relations. ConsNet [34] has leveraged word embeddings of the objects in this graph structure. Moreover, IDN [28] considered HOI as a transformation process over humans and objects.…”

“…Inspired by this approach, we use spatial configurations to guide our selfattention mechanism. Further, we combine object semantics [36,34] with spatial configurations [6] for our guidance system. Our empirical results show that this combination performs better as a guidance mechanism.…”