Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

Abstract

Dynamics of human body skeletons convey significant information for human action recognition. Conventional approaches for modeling skeletons usually rely on hand-crafted parts or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we propose a novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data. This formulation not only leads to greater expressive power but also stronger generalization capability. On two large datasets, Kinetics and NTU-RGBD, it achieves substantial improvements over mainstream methods.

Human Action Recognition From Various Data Modalities: A Review

“…This indicates that topological graph representations can be more suitable for representing the skeleton data. As a result, many GNN and GCN-based HAR methods [186], [221] have been proposed to treat the skeleton data as graph structures of edges and nodes.…”

“…Si et al [221] proposed a spatial reasoning network, followed by RNNs, to capture the high-level spatial structural and temporal dynamics of skeleton data. Shi et al [222] represented the skeleton as a directed acyclic graph to effectively shown in [186], [194], [209], [218].…”

“…More recently, GCN-based HAR has become a hot research direction [223], [224], [225], [226], [227]. Yan et al [186] exploited GCNs for skeleton-based HAR by introducing Spatial-Temporal GCNs (ST-GCNs) that can automatically learn both the spatial and temporal patterns from skeleton data, as shown in Fig. 2(c).…”

“…More specifically, the pose information was estimated from the input videos and then passed through the spatio-temporal graphs to achieve action representations with strong generalization capabilities for HAR. Since implicit joint correlations have been ignored by previous works [186], Li et al [228] further proposed an Actional-Structural GCN (AS-GCN), combining actional links and structural links into a generalized skeleton graph. The actional links were used to capture action-specific latent dependencies and the structural links were used to represent higher-order dependencies.…”

“…However, in many application scenarios, it is not convenient to deploy motion capture systems. Thus many recent works on skeleton-based HAR used skeleton data obtained from depth maps [185] or RGB videos [186].…”