Detecting human—object interaction with multi-level pairwise feature network

Abstract

Human–object interaction (HOI) detection is crucial for human-centric image understanding which aims to infer ⟨human, action, object⟩ triplets within an image. Recent studies often exploit visual features and the spatial configuration of a human–object pair in order to learn the action linking the human and object in the pair. We argue that such a paradigm of pairwise feature extraction and action inference can be applied not only at the whole human and object instance level, but also at the part level at which a body part interacts with an object, and at the semantic level by considering the semantic label of an object along with human appearance and human–object spatial configuration, to infer the action. We thus propose a multi-level pairwise feature network (PFNet) for detecting human–object interactions. The network consists of three parallel streams to characterize HOI utilizing pairwise features at the above three levels; the three streams are finally fused to give the action prediction. Extensive experiments show that our proposed PFNet outperforms other state-of-the-art methods on the V-COCO dataset and achieves comparable results to the state-of-the-art on the HICO-DET dataset.

Parallel Multi-Head Graph Attention Network (PMGAT) Model for Human-Object Interaction Detection

“…Therefore, the spatial distances must be normalized. Following the method in [59], we normalized the spatial distances between each keypoint using Equations (14).…”

“…However, as research has progressed, more emphasis on interaction details has become important. To address this, various methods have been introduced, such as attention mechanisms [9], [10], [11], [12], context information [13], [14], graph convolutional neural networks [15], [16], [17], [18], [19], body parts, and poses [20], [21], [22], [23], [24] to enhance the focus on local details within images. Specifically, the construction of context appearance features has become crucial, in addition to the visual and spatial features of humans and objects.…”

“…To address these issues, we believe that integrating human body posture and body part spatial information, as well as local facial details, into embedding a graph provides more interpretable information. While previous graph-based methods have attempted to encode human body posture and body part spatial information into node embeddings [18], [14], [43], [44], [45], insufficient attention has been paid to body parts and posture features for interacting objects. In addition, facial part information was not encoded into the nodes of the graph.…”

Semantic Recognition of Human-Object Interactions via Gaussian-Based Elliptical Modeling and Pixel-Level Labeling

“…Many HOI recognition systems have been proposed in recent years comprising of both deep learning [18,19,20] and machine learning based approaches [21]. However, in our proposed work, we have developed a machine learning based multi-vision sensors system that incorporates a semantic segmentation technique.…”

Human-object interaction detection based on graph model

“…The method based on local instance mainly analyzes the intrinsic relation between human and object from the local features such as bones, parts, and postures of the object subject. In order to extract more fine-grained information, Liu et al [4] constructed a body part-based dataset HAKE and proposed a multi-level pairwise feature network (PFNet). Zhong et al [5] proposed the glance and gaze network (GGNet), which adaptively models a set of action perception points through two steps of glance and gaze.…”