Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning

Si, Chenyang; Ya, Jing; Wang, Wei; Wang, Liang; Tan, Tieniu

doi:10.1007/978-3-030-01246-5_7

Cited by 355 publications

(269 citation statements)

References 27 publications

Supporting

Mentioning

267

Contrasting

Unclassified

Order By: Relevance

“…SYSU dataset: For the empirical evaluations, we compare our DACNN algorithm to other baselines including CNN+DPRL [28], ST-LSTM+Trust Gate [13], Dynamic Skeletons [35], LAFF(SKL) [45], SR-TSL [46], VA-LSTM [47], and GCA-LSTM [48], which includes the most recent deep learning applications (CNN, LSTM, etc.) on this dataset.…”

Section: Action Recognition Resultsmentioning

confidence: 99%

“…NTU dataset: For the NTU dataset, we evaluate DACNN in comparison to the state-of-theart methods from the literature: HBRNN-L [8], Dynamic Skeletons [35], LieNet-3Blocks [49], Part-aware LSTM [36], ST-LSTM+Trust Gate [13], CNN+LSTM2 [43], Two-Stream RNN [15], STA-LSTM [12], GCA-LSTM (stepwise) [48], Clips+CNN+MTLN [10], View invariant [14], CNN+DPRL [28], VA-LSTM [47], ST-GCN [11], CNN+LSTM [50], Two-Stream CNN [9], and SR-TSL [46]. As we can see in Table 3, DACNN did not beat SR-TSL and CNN+LSTM in recognition accuracy.…”

Section: Action Recognition Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Hosseini

Montagne

Hammer

2019

2019 IEEE International Conference on Data Mining (ICDM)

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) are deep learning frameworks which are well-known for their notable performance in classification tasks. Hence, many skeleton-based action recognition and segmentation (SBARS) algorithms benefit from them in their designs. However, a shortcoming of such applications is the general lack of spatial relationships between the input features in such data types. Besides, non-uniform temporal scalings is a common issue in skeletonbased data streams which leads to having different input sizes even within one specific action category. In this work, we propose a novel deep-aligned convolutional neural network (DACNN) to tackle the above challenges for the particular problem of SBARS. Our network is designed by introducing a new type of filters in the context of CNNs which are trained based on their alignments to the local subsequences in the inputs. These filters result in efficient predictions as well as learning interpretable patterns in the data. We empirically evaluate our framework on real-world benchmarks showing that the proposed DACNN algorithm obtains a competitive performance compared to the state-of-the-art while benefiting from a less complicated yet more interpretable model. * Preprint of the publication [1] including extended experiments, as provided by the authors. The final publication is available at https://ieeexplore.ieee.org/ networks (RNN) [8,12,13]. RNN methods can learn the temporal dynamics of the sequential data; nevertheless, they have practical shortcomings in the training of their stacked structures [14,15].Compared to RNN architectures, CNN-based methods provide more effective solutions by extracting local features from their input and finding discriminative patterns in the data [16,10]. However, regardless of the promising feature extraction capability of CNN, its specific convolutional structure is designed originally for image-based input data and primarily relies on spatial dependencies between the neighboring points. In contrast, such a direct relationship does not generally exist in skeleton-based action datasets. Although some works tried to solve this problem by using 1-dimensional filters (only for the temporal dimension), it is still not an efficient solution to this specific shortcoming of CNN-based frameworks [17].A crucial step before analyzing any motion data is the temporal segmentation phase, in which we predict the action to which each time-frame belongs. Although there exist unsupervised algorithms for temporal segmentation of motion data [18,19], they usually oversegment actions into smaller sub-actions or segment also the blank parts of the stream.Motivations: An essential group of techniques for classification of the sequential data is time-series alignment methods [20]. It is shown that via comparing each data sequence to some predefined or learned sequences, we can discriminate or segment the data samples with high accuracy [21,22]. Also, in algorithms similar to [23], finding a small distinct subsequence in the input data can reveal i...

show abstract

Section: Action Recognition Resultsmentioning

confidence: 99%

Section: Action Recognition Resultsmentioning

confidence: 99%

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Hosseini

Montagne

Hammer

2019

2019 IEEE International Conference on Data Mining (ICDM)

View full text Add to dashboard Cite

show abstract

“…In this set of experiments, we compare our model with the state-of-the-art methods for action recognition on NTU dataset in both Cross-Subject (CS) and Cross-View (CV) metrics. It should be noted that we do not compare with methods which employ extra information or prior knowledge such as joint connections for each part of body or human body structure modeling [36,48]. Table 1 reports the experimental results.…”

Section: Comparison With State-of-the-artsmentioning

confidence: 99%

Non-Local Recurrent Neural Memory for Supervised Sequence Modeling

Pei

Cao

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Typical methods for supervised sequence modeling are built upon the recurrent neural networks to capture temporal dependencies. One potential limitation of these methods is that they only model explicitly information interactions between adjacent time steps in a sequence, hence the high-order interactions between nonadjacent time steps are not fully exploited. It greatly limits the capability of modeling the long-range temporal dependencies since oneorder interactions cannot be maintained for a long term due to information dilution and gradient vanishing. To tackle this limitation, we propose the Non-local Recurrent Neural Memory (NRNM) for supervised sequence modeling, which performs non-local operations to learn full-order interactions within a sliding temporal block and models global interactions between blocks in a gated recurrent manner. Consequently, our model is able to capture the long-range dependencies. Besides, the latent high-level features contained in high-order interactions can be distilled by our model. We demonstrate the merits of our NRNM on two different tasks: action recognition and sentiment analysis. * Both authors contributed equally. † Corresponding authors.

show abstract

“…For small human action recognition datasets, deep learning methods may not give the best performance. Recent Kinect-based human action recognition algorithms are: [14,18,[20][21][22][25][26][27][28][29][30][31][32][33][34][35][36][37][38]. Research contributions.…”

Section: Introductionmentioning

confidence: 99%

A Comparative Review of Recent Kinect-Based Action Recognition Algorithms

Wang

Huynh

Koniusz

2020

IEEE Trans. on Image Process.

218

View full text Add to dashboard Cite

Video-based human action recognition is currently one of the most active research areas in computer vision. Various research studies indicate that the performance of action recognition is highly dependent on the type of features being extracted and how the actions are represented. Since the release of the Kinect camera, a large number of Kinect-based human action recognition techniques have been proposed in the literature. However, there still does not exist a thorough comparison of these Kinect-based techniques under the grouping of feature types, such as handcrafted versus deep learning features and depthbased versus skeleton-based features. In this paper, we analyze and compare ten recent Kinect-based algorithms for both crosssubject action recognition and cross-view action recognition using six benchmark datasets. In addition, we have implemented and improved some of these techniques and included their variants in the comparison. Our experiments show that the majority of methods perform better on cross-subject action recognition than cross-view action recognition, that skeleton-based features are more robust for cross-view recognition than depth-based features, and that deep learning features are suitable for large datasets.

show abstract

Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning

Cited by 355 publications

References 27 publications

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Non-Local Recurrent Neural Memory for Supervised Sequence Modeling

A Comparative Review of Recent Kinect-Based Action Recognition Algorithms

Contact Info

Product

Resources

About