Lighten

Abstract

Analyzing the interactions between humans and objects from a video includes identification of the relationships between humans and the objects present in the video. It can be thought of as a specialized version of Visual Relationship Detection, wherein one of the objects must be a human. While traditional methods formulate the problem as inference on a sequence of video segments, we present a hierarchical approach, LIGHTEN, to learn visual features to effectively capture spatio-temporal cues at multiple granularities in a video. Unlike current approaches, LIGHTEN avoids using ground truth data like depth maps or 3D human pose, thus increasing generalization across non-RGBD datasets as well. Furthermore, we achieve the same using only the visual features, instead of the

Recognizing human-object interaction in videos with knowledge guidance

“…In experiments, CAD-120 dataset 1 was used. LIGHTEN 15 and ASSIGN 17 achieve state-of-the-art of sub-activity and affordance, respectively. Therefore, we use these two networks as our baseline.…”

“…The network named LIGHTEN uses a hierarchical RNN structure to fuse inter-frame features and inter-segment features. 15 Joanna et al 16 proposed a detection method of compositional action recognition that uses Multi-Layer Perceptron to fuse spatial and temporal features. Considering the asynchronism and sparsity in HOI, Morais et al 17 proposed a hierarchical recurrent spatial-temporal graph network (ASSIGN) to automatically detect the structure of interaction.…”

Human–object interaction prediction in videos through gaze following

“…Truong and Yoshitaka (2017) refine the S-RNN by additionally considering object-object relations. Sunkesula et al (2020) further improve the model performance by applying learned visual features as the graph nodes. Instead of RNNs, Qi et al (2018) propose a Graph Parsing Network (GPN) to parse the spatio-temporal graphs of human-object interactions.…”

“…The ability to anticipate subsequent HOIs is beneficial for task planning and danger avoidance. Nevertheless, there are very few studies addressing the HOI anticipation task from the third-person view (Jain et al, 2016;Jiyang Gao and Nevatia, 2017;Truong and Yoshitaka, 2017;Sunkesula et al, 2020). However, these works are conducted on small-scale datasets and cannot be generalized to real-world applications.…”

Parallel Multi-Head Graph Attention Network (PMGAT) Model for Human-Object Interaction Detection

“…The field of HOI detection has been further enriched by recent advances in the neural heuristic analysis of video content, with the application of graph neural networks becoming increasingly compelling, from the initial generic approach evolved to a nuanced analysis that takes into account specific body parts and poses, while utilizing graph models and attentional mechanisms to improve accuracy. In recent years, the research community has introduced innovative frameworks to address multifaceted challenges that address the problem of modeling interactions at different levels of granularity [35], [36], overcoming the challenges posed by label skew [37], and the complexities of effectively integrating multimodal data [38]. These innovations have significantly advanced application areas such as scene understanding and VQA.…”