Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition

Object Knowledge Integration for Skeleton-Based Action Recognition

“…However, existing approaches mainly rely on RGB-based methods, overlooking the potential of skeleton-based modality. As a result, only a limited number of researchers have explored this modality for finegrained analysis of human-centric videos through HOI analysis [16], [15]. Previous studies have explored the use of RGB modality for object position and skeleton modality for human position analysis.…”

“…Specifically, we combine RGB-dense features with the skeleton modality to form the input representation of the self-attention model. Notably, some prior works, such as Xu et al [16], did not account for the temporal dynamics of objects in human-object interactions and treated all detected objects equally. Wang et al [15] introduced a multi-stream network with three streams, which were later fused by averaging the classification scores of each stream and not trained together.…”

“…Particularly, in the presence of proficient pre-trained RGB object detection models and pioneering datasets that encompass most of the requisite objects while skeleton data remain focused on human behavior, ultimately discerning potential actions. Certainly, there exist studies upon the same principle; however, this remains an unsettled and relatively unexplored issue, and in general cases, researchers handle the operation through several different training times, leading to amplified parameter numbers and heightened computational expenses [15], [16].…”