Temporal Attention-Augmented Graph Convolutional Network for Efficient Skeleton-Based Human Action Recognition

Abstract

Graph convolutional networks (GCNs) have been very successful in modeling non-Euclidean data structures, like sequences of body skeletons forming actions modeled as spatiotemporal graphs. Most GCN-based action recognition methods use deep feed-forward networks with high computational complexity to process all skeletons in an action. This leads to a high number of floating point operations (ranging from 16G to 100G FLOPs) to process a single sample, making their adoption in restricted computation application scenarios infeasible. In this paper, we propose a temporal attention module (TAM) for increasing the efficiency in skeleton-based action recognition by selecting the most informative skeletons of an action at the early layers of the network. We incorporate the TAM in a lightweight GCN topology to further reduce the overall number of computations. Experimental results on two benchmark datasets show that the proposed method outperforms with a large margin the baseline GCN-based method while having ×2.9 less number of computations. Moreover, it performs on par with the state-ofthe-art with up to ×9.6 less number of computations.

Skeleton Graph-Neural-Network-Based Human Action Recognition: A Survey

“…Some approaches argue to use the temporal attention to select the most informative frames or joints that are instructive in temporal dimension. N. Heidari et al [ 130 ] proposed a temporal attention module (TAM) to increase the efficiency in skeleton-based action recognition. It selects the most informative skeletons of an action, in other words, skeletons corresponding to the top highest attention values at the shallow layers of the network.…”

“…Green texts annotate the methods measured by ‘Params(M)’, and blue texts annotate ‘FLOPs (G)/Action’. The numbers around dots denote [ 13 , 14 , 50 , 54 , 55 , 57 , 59 , 62 , 73 , 77 , 80 , 82 , 85 , 86 , 89 , 110 , 116 , 117 , 121 , 127 , 130 , 135 , 141 , 183 ] respectively in ascending order. Number 94 [ 54 ] is a remarkable one, with both relatively low complexity and small model size.…”

“…Table A2 summarizes the action list and methods performed on each dataset. The most popular compared methods across datasets are [ 13 , 14 , 50 , 54 , 55 , 57 , 59 , 62 , 73 , 77 , 80 , 82 , 85 , 86 , 89 , 110 , 116 , 117 , 121 , 127 , 130 , 135 , 141 , 183 ], among which, Ref. [ 54 ] is a remarkable one on NTU RGB+D cross-subject, with both low complexity and small model size but good performance.…”

Non-local Graph Convolutional Network for joint Activity Recognition and Motion Prediction

“…Recently, GNN or GCN have been demonstrated to be a more principled and effective choice for parsing the graph structure of skeleton data [10], which enables innercorrelation capture without segmentation of the whole body. Another example, TA-GCN [11], proposes to use an attention module in a GCN-based spatial-temporal model to extract more useful predictive features from graph data. However these works are focused on only action recognition so that it could not exploit available motion data to enhance the recognition ability.…”