A Graph-Based Approach to Recognizing Complex Human Object Interactions in Sequential Data

Abstract

The critical task of recognizing human–object interactions (HOI) finds its application in the domains of surveillance, security, healthcare, assisted living, rehabilitation, sports, and online learning. This has led to the development of various HOI recognition systems in the recent past. Thus, the purpose of this study is to develop a novel graph-based solution for this purpose. In particular, the proposed system takes sequential data as input and recognizes the HOI interaction being performed in it. That is, first of all, the system pre-processes the input data by adjusting the contrast and smoothing the incoming image frames. Then, it locates the human and object through image segmentation. Based on this, 12 key body parts are identified from the extracted human silhouette through a graph-based image skeletonization technique called image foresting transform (IFT). Then, three types of features are extracted: full-body feature, point-based features, and scene features. The next step involves optimizing the different features using isometric mapping (ISOMAP). Lastly, the optimized feature vector is fed to a graph convolution network (GCN) which performs the HOI classification. The performance of the proposed system was validated using three benchmark datasets, namely, Olympic Sports, MSR Daily Activity 3D, and D3D-HOI. The results showed that this model outperforms the existing state-of-the-art models by achieving a mean accuracy of 94.1% with the Olympic Sports, 93.2% with the MSR Daily Activity 3D, and 89.6% with the D3D-HOI datasets.

Ambient Assisted Living: Scoping Review of Artificial Intelligence Models, Domains, Technology, and Concerns

“…The distribution of AI classes shows that semisupervised learning models had prevailed since the 2010s, with an irregular growing trend until 2017, when they compensated for the lack of a sufficient amount of labeled data for particular inputs. Concrete examples include clustering for physical activity recognition [ 37 ], finding relevant input features for improving activity recognition [ 27 ], and detecting user-object interactions from sequences of images [ 143 ].…”

“…Semisupervised and unsupervised learning classes dominate the intelligent AAL landscape. Their prevalence is because of an increase in the variety of the health and living domain and the gradual appearance of labeled input data describing related ADL and IADL the learning aimed to support [ 37 , 38 , 143 ]. Supervised approaches have been used for classification tasks [ 41 ].…”

Parallel Multi-Head Graph Attention Network (PMGAT) Model for Human-Object Interaction Detection

“…To address these issues, we believe that integrating human body posture and body part spatial information, as well as local facial details, into embedding a graph provides more interpretable information. While previous graph-based methods have attempted to encode human body posture and body part spatial information into node embeddings [18], [14], [43], [44], [45], insufficient attention has been paid to body parts and posture features for interacting objects. In addition, facial part information was not encoded into the nodes of the graph.…”