Spatial-Temporal Attention Res-TCN for Skeleton-Based Dynamic Hand Gesture Recognition

Hou, Jingxuan; Wang, Guijin; Chen, Xinghao; Xue, Jing-Hao; Zhu, Rui; Yang, Huazhong

doi:10.1007/978-3-030-11024-6_18

Cited by 104 publications

(63 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DSTM [14] is excluded from this experiment for not giving the reference data and experiment result. HG-GCN improves 0.3% accuracy over STA-Res-TCN [12] for the complicated 28 gestures setting, which shows the advantage of the proposed method on recognizing fine gestures. As for the comparison with other methods, HG-GCN shows its advantages obviously.…”

Section: Resultsmentioning

confidence: 82%

“…3a: four joints for each finger, one joint (1) for the palm, and one joint (0) for the wrist. The thumb contains joints (5, 4, 3, 2), the index finger contains joints (9, 8, 7, 6), the middle finger contains joints (13,12,11,10), the ring finger contains joints (17,16,15,14), and the pinkie contains joints (21,20,19,18). Yan et al linked the 18 joints of the human body skeleton with 17 edges from head to foot in order.…”

Section: Hand Gesture Graph Convnetmentioning

confidence: 99%

“…The method in [9] treated the skeleton joints as a pseudoimage and that in [10] treated skeleton joints as a vector sequence, but the connectivity of joints was not explicitly considered. Hou et al [12] designed an end-to-end spatialtemporal attention residual temporal convolutional network (STA-Res-TCN) which modifies temporal convolutional networks [13] for skeleton-based dynamic hand gesture recognition. STA-Res-TCN models the connectivity between joints by multiplying the output of an additional branch with the original branch, which forms the soft attention mechanism.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition

Xiang

et al. 2019

J Image Video Proc.

View full text Add to dashboard Cite

Hand gesture recognition methods play an important role in human-computer interaction. Among these methods are skeleton-based recognition techniques that seem to be promising. In literature, several methods have been proposed to recognize hand gestures with skeletons. One problem with these methods is that they consider little the connectivity between the joints of a skeleton, constructing simple graphs for skeleton connectivity. Observing this, we built a new model of hand skeletons by adding three types of edges in the graph to finely describe the linkage action of joints. Then, an end-to-end deep neural network, hand gesture graph convolutional network, is presented in which the convolution is conducted only on linked skeleton joints. Since the training dataset is relatively small, this work proposes expanding the coordinate dimensionality so as to let models learn more semantic features. Furthermore, relative coordinates are employed to help hand gesture graph convolutional network learn the feature representation independent of the random starting positions of actions. The proposed method is validated on two challenging datasets, and the experimental results show that it outperforms the state-of-the-art methods. Furthermore, it is relatively lightweight in practice for hand skeleton-based gesture recognition.

show abstract

Section: Resultsmentioning

confidence: 82%

Section: Hand Gesture Graph Convnetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition

Xiang

et al. 2019

J Image Video Proc.

View full text Add to dashboard Cite

show abstract

“…For skeleton-based action recognition, two types of input features are commonly used: the geometric feature [18], [22] and the Cartesian coordinate feature [31], [32], [34], [6], [7]. The Cartesian coordinate feature is variant to locations and viewpoints.…”

Section: A Modeling Location-viewpoint Invariant Feature By Joint Comentioning

confidence: 99%

Make Skeleton-based Action Recognition Model Smaller, Faster and Better

Yang

Wu²,

Sakti

et al. 2019

Proceedings of the ACM Multimedia Asia

141

View full text Add to dashboard Cite

Although skeleton-based action recognition has achieved great success in recent years, most of the existing methods may suffer from a large model size and slow execution speed. To alleviate this issue, we analyze skeleton sequence properties to propose a Double-feature Double-motion Network (DD-Net) for skeleton-based action recognition. By using a lightweight network structure (i.e., 0.15 million parameters), DD-Net can reach a super fast speed, as 3,500 FPS on one GPU, or, 2,000 FPS on one CPU. By employing robust features, DD-Net achieves the state-of-the-art performance on our experiment datasets: SHREC (i.e., hand actions) and JHMDB (i.e., body actions). Our code is on https://github.com/fandulu/DD-Net.

show abstract

“…However, these handcrafted features cannot model the spatial and temporal information effectively. Recently, several deep learning-based works [4,15,16] are proposed, which feed hand skeleton sequence into LSTM or CNN networks for hand gesture recognition.…”

Section: Introductionmentioning

confidence: 99%

STA-GCN: two-stream graph convolutional network with spatial–temporal attention for hand gesture recognition

et al. 2020

View full text Add to dashboard Cite

Skeleton-based hand gesture recognition is an active research topic in computer graphics and computer vision and has a wide range of applications in VR/AR and robotics. Although the spatial-temporal graph convolutional network has been successfully used in skeleton-based hand gesture recognition, these works often use a fixed spatial graph according to the hand skeleton tree or use a fixed graph on the temporal dimension, which may not be optimal for hand gesture recognition. In this paper, we propose a two-stream graph attention convolutional network with spatial-temporal attention for hand gesture recognition. We adopt pose stream and motion stream as the two input streams for our network. In pose stream, we use the joint in each frame as the input; In motion stream, we use the joint offsets between neighboring frames as the input. We propose a new temporal graph attention module to model the temporal dependency and also use a spatial graph attention module to construct dynamic skeleton graph. For each stream, we adopt graph convolutional network with spatial-temporal attention to extract the features. Then, we concatenate the feature of the pose stream and motion stream for gesture recognition. We achieve the competitive performance on the main hand gesture recognition benchmark datasets, which demonstrates the effectiveness of our method.

show abstract

Spatial-Temporal Attention Res-TCN for Skeleton-Based Dynamic Hand Gesture Recognition

Cited by 104 publications

References 26 publications

Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition

Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition

Make Skeleton-based Action Recognition Model Smaller, Faster and Better

STA-GCN: two-stream graph convolutional network with spatial–temporal attention for hand gesture recognition

Contact Info

Product

Resources

About