Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates

Liu, Jun; Shahroudy, Amir; Xu, Dong; Kot, Alex C.; Wang, Gang

doi:10.1109/tpami.2017.2771306

Cited by 441 publications

(257 citation statements)

References 93 publications

(99 reference statements)

Supporting

Mentioning

257

Contrasting

Order By: Relevance

“…SYSU dataset: For the empirical evaluations, we compare our DACNN algorithm to other baselines including CNN+DPRL [28], ST-LSTM+Trust Gate [13], Dynamic Skeletons [35], LAFF(SKL) [45], SR-TSL [46], VA-LSTM [47], and GCA-LSTM [48], which includes the most recent deep learning applications (CNN, LSTM, etc.) on this dataset.…”

Section: Action Recognition Resultsmentioning

confidence: 99%

“…* Preprint of the publication [1] including extended experiments, as provided by the authors. The final publication is available at https://ieeexplore.ieee.org/ networks (RNN) [8,12,13]. RNN methods can learn the temporal dynamics of the sequential data; nevertheless, they have practical shortcomings in the training of their stacked structures [14,15].Compared to RNN architectures, CNN-based methods provide more effective solutions by extracting local features from their input and finding discriminative patterns in the data [16,10].…”

mentioning

confidence: 99%

See 1 more Smart Citation

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Hosseini

Montagne

Hammer

2019

2019 IEEE International Conference on Data Mining (ICDM)

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) are deep learning frameworks which are well-known for their notable performance in classification tasks. Hence, many skeleton-based action recognition and segmentation (SBARS) algorithms benefit from them in their designs. However, a shortcoming of such applications is the general lack of spatial relationships between the input features in such data types. Besides, non-uniform temporal scalings is a common issue in skeletonbased data streams which leads to having different input sizes even within one specific action category. In this work, we propose a novel deep-aligned convolutional neural network (DACNN) to tackle the above challenges for the particular problem of SBARS. Our network is designed by introducing a new type of filters in the context of CNNs which are trained based on their alignments to the local subsequences in the inputs. These filters result in efficient predictions as well as learning interpretable patterns in the data. We empirically evaluate our framework on real-world benchmarks showing that the proposed DACNN algorithm obtains a competitive performance compared to the state-of-the-art while benefiting from a less complicated yet more interpretable model. * Preprint of the publication [1] including extended experiments, as provided by the authors. The final publication is available at https://ieeexplore.ieee.org/ networks (RNN) [8,12,13]. RNN methods can learn the temporal dynamics of the sequential data; nevertheless, they have practical shortcomings in the training of their stacked structures [14,15].Compared to RNN architectures, CNN-based methods provide more effective solutions by extracting local features from their input and finding discriminative patterns in the data [16,10]. However, regardless of the promising feature extraction capability of CNN, its specific convolutional structure is designed originally for image-based input data and primarily relies on spatial dependencies between the neighboring points. In contrast, such a direct relationship does not generally exist in skeleton-based action datasets. Although some works tried to solve this problem by using 1-dimensional filters (only for the temporal dimension), it is still not an efficient solution to this specific shortcoming of CNN-based frameworks [17].A crucial step before analyzing any motion data is the temporal segmentation phase, in which we predict the action to which each time-frame belongs. Although there exist unsupervised algorithms for temporal segmentation of motion data [18,19], they usually oversegment actions into smaller sub-actions or segment also the blank parts of the stream.Motivations: An essential group of techniques for classification of the sequential data is time-series alignment methods [20]. It is shown that via comparing each data sequence to some predefined or learned sequences, we can discriminate or segment the data samples with high accuracy [21,22]. Also, in algorithms similar to [23], finding a small distinct subsequence in the input data can reveal i...

show abstract

Section: Action Recognition Resultsmentioning

confidence: 99%

mentioning

confidence: 99%

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Hosseini

Montagne

Hammer

2019

2019 IEEE International Conference on Data Mining (ICDM)

View full text Add to dashboard Cite

show abstract

“…Before the raw data can be input int the action recognition module, human key-point coordinates must be generated using the pose estimation technique. Using of human key-point coordinates to train the action recognition module will help to reduce the background clutter [37,38]. Also, it will reduce the computational complexity as compared to using the entire image/video to train the module [39,40].…”

Section: Data Pre-processingmentioning

confidence: 99%

Human Action Performance Using Deep Neuro-Fuzzy Recurrent Attention Model

et al. 2020

View full text Add to dashboard Cite

A great number of computer vision publications have focused on distinguishing between human action recognition and classification rather than the intensity of actions performed. Indexing the intensity which determines the performance of human actions is a challenging task due to the uncertainty and information deficiency that exists in the video inputs. To remedy this uncertainty, in this paper we coupled fuzzy logic rules with the neural-based action recognition model to rate the intensity of a human action as intense or mild. In our approach, we used a Spatio-Temporal LSTM to generate the weights of the fuzzy-logic model, and then demonstrate through experiments that indexing of the action intensity is possible. We analyzed the integrated model by applying it to videos of human actions with different action intensities and were able to achieve an accuracy of 89.16% on our intensity indexing generated dataset. The integrated model demonstrates the ability of a neuro-fuzzy inference module to effectively estimate the intensity index of human actions.

show abstract

“…They proposed an extension of long short term memory (LSTM) model which leverages group motion of several body joints to recognize human activity from skeletal data. A different adaptation of the LSTM model was proposed by Liu et al [14] where spatial interaction among joints was considered in addition to the temporal dynamics. Veeriah et al [15] proposed a LSTM network to capture the salient motion pattern of body joints.…”

Section: Literature Reviewmentioning

confidence: 99%

Sign Language Recognition Analysis using Multimodal Data

Hosain

Santhalingam

Pathak

et al. 2019

2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA)

View full text Add to dashboard Cite

Voice-controlled personal and home assistants (such as the Amazon Echo and Apple Siri) are becoming increasingly popular for a variety of applications. However, the benefits of these technologies are not readily accessible to Deaf or Hard-of-Hearing (DHH) users. The objective of this study is to develop and evaluate a sign recognition system using multiple modalities that can be used by DHH signers to interact with voice-controlled devices. With the advancement of depth sensors, skeletal data is used for applications like video analysis and activity recognition. Despite having similarity with the well-studied human activity recognition, the use of 3D skeleton data in sign language recognition is rare. This is because unlike activity recognition, sign language is mostly dependent on hand shape pattern. In this work, we investigate the feasibility of using skeletal and RGB video data for sign language recognition using a combination of different deep learning architectures. We validate our results on a large-scale American Sign Language (ASL) dataset of 12 users and 13107 samples across 51 signs. It is named as GMU-ASL51. 1 We collected the dataset over 6 months and it will be publicly released in the hope of spurring further machine learning research towards providing improved accessibility for digital assistants. 2

show abstract

Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates

Cited by 441 publications

References 93 publications

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Deep-Aligned Convolutional Neural Network for Skeleton-Based Action Recognition and Segmentation

Human Action Performance Using Deep Neuro-Fuzzy Recurrent Attention Model

Sign Language Recognition Analysis using Multimodal Data

Contact Info

Product

Resources

About