Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition

Abstract

Action recognition with skeleton data has recently attracted much attention in computer vision. Previous studies are mostly based on fixed skeleton graphs, only capturing local physical dependencies among joints, which may miss implicit joint correlations. To capture richer dependencies, we introduce an encoder-decoder structure, called A-link inference module, to capture action-specific latent dependencies, i.e. actional links, directly from actions. We also extend the existing skeleton graphs to represent higherorder dependencies, i.e. structural links. Combing the two types of links into a generalized skeleton graph, we further propose the actional-structural graph convolution network (AS-GCN), which stacks actional-structural graph convolution and temporal convolution as a basic building block, to learn both spatial and temporal features for action recognition. A future pose prediction head is added in parallel to the recognition head to help capture more detailed action patterns through self-supervision. We validate AS-GCN in action recognition using two skeleton data sets, NTU-RGB+D and Kinetics. The proposed AS-GCN achieves consistently large improvement compared to the state-of-the-art methods. As a side product, AS-GCN also shows promising results for future pose prediction. Our code is available at https://github.com/limaosen0/AS-GCN . 1

Action Recognition Based on the Fusion of Graph Convolutional Networks with High Order Features

“…The results of the comparison are shown in Tables 5 and 6. The methods used for comparison include the handcraft-feature-based methods [33], RNN-based methods [28,29,34,35], CNN-based methods [36,37], and GCN-based methods [6][7][8][9][10]. From Table 5, we can see that our proposed method achieves the best performances of 96.8% and 91.7% in terms of two criteria on the NTU-RGBD dataset.…”

“…ST-GCN (2018) [6] 81.5 88.3 AS-GCN(2018) [9] 86.8 94.2 PB-GCN (2018) [8] 87.5 93.2 2s-AGCN(2019) [7] 88.5 95.1 AGC-LSTM(2019) [10] 89.2 95.0 ours 91.7 96.8 Table 6. The results of different methods, which are designed for 3D human activity analysis, using the cross-subject and cross-setup evaluation criteria on the NTU RGB+D 120 dataset.…”

“…Therefore, graph-based neural networks have been used for action recognition instead of the traditional CNN networks because of the successful performance. Some graph-based neural networks [6][7][8][9][10] are dedicated to learning both spatial and temporal features for action recognition. Meanwhile, they focus on capturing the hidden relationships among vertices in space.…”

Enhanced Spatial and Extended Temporal Graph Convolutional Network for Skeleton-Based Action Recognition

“…Based on this judgment, Yan et al [ 23 ] proposed a spatial-temporal graph convolutional network (ST-GCN) representing human joints as vertices and the bones as edges. The ST-GCN improves the accuracy of action recognition to a new level, and substantial ST-GCNs are subsequently proposed based on it [ 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ]. However, there are still two problems to be addressed in these methods.…”

Global Co-Occurrence Feature and Local Spatial Feature Learning for Skeleton-Based Action Recognition

“…(1) Because ST-GCN [ 4 ] may not adequately capture the dependency between far-apart joints [ 5 ], it is unable to effectively extract the global co-occurrence features of actions. (2) Since convolution cannot consider the relationship between each vertex and its surrounding vertices, these related works [ 4 , 6 , 7 ] may not effectively obtain the spatial features composed of adjacent vertices. (3) These works [ 4 , 6 , 7 ] expand the number of channels per-vertex as the number of network layers increases.…”