An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition

Si, Chenyang; Chen, Wentao; Wang, Wei; Wang, Liang; Tan, Tieniu

doi:10.1109/cvpr.2019.00132

Cited by 786 publications

(461 citation statements)

References 36 publications

Supporting

Mentioning

427

Contrasting

Order By: Relevance

“…To reason between different instances in visual question answering tasks, Xiong et al [47] proposed a graph matching module for investigating such relation. In sketch-based action recognition, several works showed that modeling interactions of sketch joint can achieve excellent performance [25], [26], [48]. Here, we show that explicitly exploiting the various and multi-scale temporal relations in videos can boost activity recognition accuracy.…”

Section: Related Workmentioning

confidence: 56%

Temporal Reasoning Graph for Activity Recognition

Zhang

Shen

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Despite great success has been achieved in activity analysis, it still has many challenges. Most existing work in activity recognition pay more attention to design efficient architecture or video sampling strategy. However, due to the property of fine-grained action and long term structure in video, activity recognition is expected to reason temporal relation between video sequences. In this paper, we propose an efficient temporal reasoning graph (TRG) to simultaneously capture the appearance features and temporal relation between video sequences at multiple time scales. Specifically, we construct learnable temporal relation graphs to explore temporal relation on the multi-scale range. Additionally, to facilitate multi-scale temporal relation extraction, we design a multihead temporal adjacent matrix to represent multi-kinds of temporal relations. Eventually, a multi-head temporal relation aggregator is proposed to extract the semantic meaning of those features convolving through the graphs. Extensive experiments are performed on widely-used large-scale datasets, such as Something-Something and Charades, and the results show that our model can achieve state-of-the-art performance. Further analysis shows that temporal relation reasoning with our TRG can extract discriminative features for activity recognition.

show abstract

Section: Related Workmentioning

confidence: 56%

Temporal Reasoning Graph for Activity Recognition

Zhang

Shen

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

show abstract

“…Kalpit divided the skeleton graph into four subgraphs with joints shared across them and taught a recognition model using a part-based graph convolutional network [8]. AGC-LSTM [10] can not only capture features in spatial configuration and temporal dynamics but also explore the co-occurrence relationship between spatial and temporal domains.…”

Section: Skeleton-based Methodsmentioning

confidence: 99%

“…Therefore, graph-based neural networks have been used for action recognition instead of the traditional CNN networks because of the successful performance. Some graph-based neural networks [6][7][8][9][10] are dedicated to learning both spatial and temporal features for action recognition. Meanwhile, they focus on capturing the hidden relationships among vertices in space.…”

Section: Introductmentioning

confidence: 99%

Action Recognition Based on the Fusion of Graph Convolutional Networks with High Order Features

et al. 2020

View full text Add to dashboard Cite

Skeleton-based action recognition is a widely used task in action related research because of its clear features and the invariance of human appearances and illumination. Furthermore, it can also effectively improve the robustness of the action recognition. Graph convolutional networks have been implemented on those skeletal data to recognize actions. Recent studies have shown that the graph convolutional neural network works well in the action recognition task using spatial and temporal features of skeleton data. The prevalent methods to extract the spatial and temporal features purely rely on a deep network to learn from primitive 3D position. In this paper, we propose a novel action recognition method applying high-order spatial and temporal features from skeleton data, such as velocity features, acceleration features, and relative distance between 3D joints. Meanwhile, a method of multi-stream feature fusion is adopted to fuse these high-order features we proposed. Extensive experiments on Two large and challenging datasets, NTU-RGBD and NTU-RGBD-120, indicate that our model achieves the state-of-the-art performance.

show abstract

“…Apart from the CNN-based approaches [20,77], Recurrent Neural Networks (RNNs) have also been popular [18,27,35,38,65,[67][68][69][70]. Since Long Short-term Memory (LSTM) [78] can model temporal dependencies as RNNs and even capture the co-occurrences of human joints, LSTM networks have also been a popular choice in human action recognition [31,33,36,37,73]. For instance, Zhu et al [69] presented an end-to-end deep LSTM network with a dropout step, Shahroudy et al [18] proposed a Part-aware Long Short-term Memory (P-LSTM) network to learn the long-term patterns of the 3D trajectories for each grouped body part, and Liu et al [36] introduced the use of trust gates in their spatio-temporal LSTM architecture.…”

Section: Related Workmentioning

confidence: 99%

A Comparative Review of Recent Kinect-Based Action Recognition Algorithms

Wang

Huynh

Koniusz

2020

IEEE Trans. on Image Process.

218

View full text Add to dashboard Cite

Video-based human action recognition is currently one of the most active research areas in computer vision. Various research studies indicate that the performance of action recognition is highly dependent on the type of features being extracted and how the actions are represented. Since the release of the Kinect camera, a large number of Kinect-based human action recognition techniques have been proposed in the literature. However, there still does not exist a thorough comparison of these Kinect-based techniques under the grouping of feature types, such as handcrafted versus deep learning features and depthbased versus skeleton-based features. In this paper, we analyze and compare ten recent Kinect-based algorithms for both crosssubject action recognition and cross-view action recognition using six benchmark datasets. In addition, we have implemented and improved some of these techniques and included their variants in the comparison. Our experiments show that the majority of methods perform better on cross-subject action recognition than cross-view action recognition, that skeleton-based features are more robust for cross-view recognition than depth-based features, and that deep learning features are suitable for large datasets.

show abstract

An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition

Cited by 786 publications

References 36 publications

Temporal Reasoning Graph for Activity Recognition

Temporal Reasoning Graph for Activity Recognition

Action Recognition Based on the Fusion of Graph Convolutional Networks with High Order Features

A Comparative Review of Recent Kinect-Based Action Recognition Algorithms

Contact Info

Product

Resources

About