Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature

Liu, Zhipeng; Chai, Xiujuan; Liu, Zhuang; Chen, Xilin

doi:10.1109/iccvw.2017.361

Cited by 62 publications

(62 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But this approach was hard to have universal applicability, because some hand motion information could not be converted into static image. In [27], it presented a spotting-recognition framework for large-scale continuous gesture recognition. Camgoz et al [28] used the sequence-tosequence to learn the recognition problems of sign language.…”

Section: B Methods Based On Video Sequencementioning

confidence: 99%

Dynamic Sign Language Recognition Based on Video Sequence With BLSTM-3D Residual Networks

et al. 2019

View full text Add to dashboard Cite

Sign language recognition aims to recognize meaningful movements of hand gestures and is a significant solution in intelligent communication between the deaf community and hearing societies. However, until now, the current dynamic sign language recognition methods still have some drawbacks with difficulties of recognizing complex hand gestures, low recognition accuracy for most dynamic sign language recognition, and potential problems in larger video sequence data training. In order to solve these issues, this paper presents a multimodal dynamic sign language recognition method based on a deep 3-dimensional residual ConvNet and bi-directional LSTM networks, which is named as BLSTM-3D residual network (B3D ResNet). This method consists of three main parts. First, the hand object is localized in the video frames in order to reduce the time and space complexity of network calculation. Then, the B3D ResNet automatically extracts the spatiotemporal features from the video sequences and establishes an intermediate score corresponding to each action in the video sequence after feature analysis. Finally, by classifying the video sequences, the dynamic sign language is accurately identified. The experiment is conducted on test datasets, including DEVISIGN_D dataset and SLR_Dataset. The results show that the proposed method can obtain state-of-the-art recognition accuracy (89.8% on the DEVISIGN_D dataset and 86.9% on SLR_Dataset). In addition, the B3D ResNet can effectively recognize complex hand gestures through larger video sequence data, and obtain high recognition accuracy for 500 vocabularies from Chinese hand sign language.

show abstract

Section: B Methods Based On Video Sequencementioning

confidence: 99%

Dynamic Sign Language Recognition Based on Video Sequence With BLSTM-3D Residual Networks

et al. 2019

View full text Add to dashboard Cite

show abstract

“…The multiple cues of sign language can be separated into categories of multi-modality and multi-semantic. Early works about multi-modality utilize physical sensors to collect the 3D space information, such as depth and infrared maps (Molchanov et al 2016;Liu et al 2017). With the development of flow estimation, Cui et al (Cui, Liu, and Zhang 2019) explore the multi-modality fusion of RGB and optical flow and achieve state-of-the-art performance on PHOENIX-2014 database.…”

Section: Related Workmentioning

confidence: 99%

Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition

Zhou

et al. 2020

AAAI

149

View full text Add to dashboard Cite

Despite the recent success of deep learning in continuous sign language recognition (CSLR), deep models typically focus on the most discriminative features, ignoring other potentially non-trivial and informative contents. Such characteristic heavily constrains their capability to learn implicit visual grammars behind the collaboration of different visual cues (i,e., hand shape, facial expression and body posture). By injecting multi-cue learning into neural network design, we propose a spatial-temporal multi-cue (STMC) network to solve the vision-based sequence learning problem. Our STMC network consists of a spatial multi-cue (SMC) module and a temporal multi-cue (TMC) module. The SMC module is dedicated to spatial representation and explicitly decomposes visual features of different cues with the aid of a self-contained pose estimation branch. The TMC module models temporal correlations along two parallel paths, i.e., intra-cue and inter-cue, which aims to preserve the uniqueness and explore the collaboration of multiple cues. Finally, we design a joint optimization strategy to achieve the end-to-end sequence learning of the STMC network. To validate the effectiveness, we perform experiments on three large-scale CSLR benchmarks: PHOENIX-2014, CSL and PHOENIX-2014-T. Experimental results demonstrate that the proposed method achieves new state-of-the-art performance on all three benchmarks.

show abstract

“…Due to the basic status of convolutional neural networks (CNN) in deep learning networks, some research teams have conducted a series of CNN-based isolated sign language recognition studies since 2013 [ 6 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 ]. Based on CNN recognition, the algorithm can be optimized by adding multi-modal data (including depth, skeleton, key points of the human body, etc.…”

Section: Related Workmentioning

confidence: 99%

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Lü

2021

Sensors

View full text Add to dashboard Cite

Sign language is the most important way of communication for hearing-impaired people. Research on sign language recognition can help normal people understand sign language. We reviewed the classic methods of sign language recognition, and the recognition accuracy is not high enough because of redundant information, human finger occlusion, motion blurring, the diversified signing styles of different people, and so on. To overcome these shortcomings, we propose a multi-scale and dual sign language recognition Network (SLR-Net) based on a graph convolutional network (GCN). The original input data was RGB videos. We first extracted the skeleton data from them and then used the skeleton data for sign language recognition. SLR-Net is mainly composed of three sub-modules: multi-scale attention network (MSA), multi-scale spatiotemporal attention network (MSSTA) and attention enhanced temporal convolution network (ATCN). MSA allows the GCN to learn the dependencies between long-distance vertices; MSSTA can directly learn the spatiotemporal features; ATCN allows the GCN network to better learn the long temporal dependencies. The three different attention mechanisms, multi-scale attention mechanism, spatiotemporal attention mechanism, and temporal attention mechanism, are proposed to further improve the robustness and accuracy. Besides, a keyframe extraction algorithm is proposed, which can greatly improve efficiency by sacrificing a little accuracy. Experimental results showed that our method can reach 98.08% accuracy rate in the CSL-500 dataset with a 500-word vocabulary. Even on the challenging dataset DEVISIGN-L with a 2000-word vocabulary, it also reached a 64.57% accuracy rate, outperforming other state-of-the-art sign language recognition methods.

show abstract

Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature

Cited by 62 publications

References 16 publications

Dynamic Sign Language Recognition Based on Video Sequence With BLSTM-3D Residual Networks

Dynamic Sign Language Recognition Based on Video Sequence With BLSTM-3D Residual Networks

Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Contact Info

Product

Resources

About