SequentialPointNet: A strong frame-level parallel point cloud sequence network for 3D action recognition

Abstract

Point cloud sequences of 3D human actions exhibit unordered intra-frame spatial information and ordered interframe temporal information. In order to capture the spatiotemporal structures of the point cloud sequences, cross-frame spatio-temporal local neighborhoods around the centroids are usually constructed. However, the computationally expensive construction procedure of spatio-temporal local neighborhoods severely limits the parallelism of models. Moreover, it is unreasonable to treat spatial and temporal information equally in spatio-temporal local learning, because human actions are complicated along the spatial dimensions and simple along the temporal dimension. In this paper, to avoid spatio-temporal local encoding, we propose a strong parallelized point cloud sequence network referred to as SequentialPointNet for 3D action recognition. SequentialPointNet is composed of two serial modules, i.e., an intra-frame appearance encoding module and an inter-frame motion encoding module. For modeling the strong spatial structures of human actions, each point cloud frame is processed in parallel in the intra-frame appearance encoding module and the feature vector of each frame is output to form a feature vector sequence that characterizes static appearance changes along the temporal dimension. For modeling the weak temporal changes of human actions, in the inter-frame motion encoding module, the temporal position encoding and the hierarchical pyramid pooling strategy are implemented on the feature vector sequence. In addition, in order to better explore spatio-temporal content, multiple level features of human movements are aggregated before performing the end-to-end 3D action recognition. Extensive experiments conducted on three public datasets show that SequentialPointNet outperforms stateof-the-art approaches. SequentialPointNet's code is available at https://github.com/XingLi1012/SequentialPointNet.git.

Motion-Driven Spatial and Temporal Adaptive High-Resolution Graph Convolutional Networks for Skeleton-Based Action Recognition

“…To the best of our knowledge, so far, no GCN-based method has been evaluated on it. We compare our methods with two publicly available classical GCN-based methods (i.e., ST-GCN [25] and 2s-AGCN [26]), and the SOTA CNN-based methods (e.g., RIAC-LSTM [23], SPMFs [78]), point cloud-based methods (e.g., SequentialPointNet [56], P4Transformer [55]), and…”

“…Year Accuracy(%) ST-GCN [25] 2018 83.27 SPMFs [78] 2018 98.05 2s-AGCN [26] 2019 88.36 MeteorNet [79] 2019 88.5 UnifiedDeep [80] 2019 97.98 Movement polygon [13] 2020 94.13 P4Transformer [55] 2021 90.94 PSTNet [81] 2021 91.2 MMDNN [82] 2021 91.94 RIAC-LSTM [23] 2021 98.06 Complex Network+LSTM [83] 2022 90.7 SequentialPointNet [56] 2022 91.94 2s-MS&TA-HGCN-FC(ours) 90.54 4s-MS&TA-HGCN-FC(ours) 92. 73 depth-based methods (e.g., MMDNN [82]).…”

“…Each action was performed three times by every subject. In the experiment, we follow the recent literature [23], [55], [56] and use the cross-subject evaluation protocol where half of the subjects are used for training and the rest are used for testing. We use the pretrained GCN models on NTU-RGB+D datasets to funtune for evaluation because of the insufficient training samples of this dataset.…”

PointDMIG: A Dynamic Motion-Informed Graph Neural Network for 3D Action Recognition

“…These pieces of information enable a better representation of the 3D structure and characteristics of scenes and objects. Therefore, recent works has begun to focus on 3D action recognition based on point clouds [11][12][13][14][15]. The primary principle is to convert depth videos into point cloud to represent the geometric structure and distance information of object surfaces.…”

“…Nevertheless, improper selection of the time step may cause the loss of some spatial information. Li et al [14,15] propose that human behaviors exhibit strong spatial structures and weak temporal changes. They independently encode the spatial and temporal dimensions of point cloud sequences without constructing cross-frame local neighborhoods, effectively capturing the spatial structure information of point cloud sequences.…”