Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition

Zhou, Hao; Zhou, Wengang; Zhou, Yun; Li, Houqiang

doi:10.1609/aaai.v34i07.7001

Cited by 150 publications

(107 citation statements)

References 55 publications

(104 reference statements)

Supporting

Mentioning

107

Contrasting

Order By: Relevance

“…Then, a translation system translates the recognized glosses into spoken language. Recent work (Orbay and Akarun, 2020;Zhou et al, 2020) has addressed the first step, but there has been none improving the translation system. This paper aims to fill this research gap by leveraging recent success in Neural Machine Translation (NMT), namely Transformers.…”

Section: Introductionmentioning

confidence: 99%

Better Sign Language Translation with STMC-Transformer

Yin

Read

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Sign Language Translation (SLT) first uses a Sign Language Recognition (SLR) system to extract sign language glosses from videos. Then, a translation system generates spoken language translations from the sign language glosses. This paper focuses on the translation system and introduces the STMC-Transformer which improves on the current state-of-the-art by over 5 and 7 BLEU respectively on gloss-to-text and video-to-text translation of the PHOENIX-Weather 2014T dataset. On the ASLG-PC12 corpus, we report an increase of over 16 BLEU.We also demonstrate the problem in current methods that rely on gloss supervision. The videoto-text translation of our STMC-Transformer outperforms translation of GT glosses. This contradicts previous claims that GT gloss translation acts as an upper bound for SLT performance and reveals that glosses are an inefficient representation of sign language. For future SLT research, we therefore suggest an end-to-end training of the recognition and translation models, or using a different sign language annotation scheme. *

show abstract

Section: Introductionmentioning

confidence: 99%

Better Sign Language Translation with STMC-Transformer

Yin

Read

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…The whole architecture was optimized iteratively in two stages using CTC loss and cross-entropy loss with pseudo-labels. In [ 45 ], the authors proposed a method for exploiting not only RGB data, but also information from multiple cues, such as the pose, hands, and face of the signer, aiming to find correlations between the different cues.…”

Section: Related Workmentioning

confidence: 99%

“…In Table 3 , the proposed method is compared against several state-of-the-art methods using only RGB data (for fair comparison, methods based on multi-cue information, e.g., [ 45 ], are not included in Table 3 ) on the RWTH-Phoenix-Weather-2014 dataset. SLRGAN achieved a WER of

on the validation set and

on the test set.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network

Papastratis

Dimitropoulos

Daras

2021

Sensors

View full text Add to dashboard Cite

Continuous sign language recognition is a weakly supervised task dealing with the identification of continuous sign gestures from video sequences, without any prior knowledge about the temporal boundaries between consecutive signs. Most of the existing methods focus mainly on the extraction of spatio-temporal visual features without exploiting text or contextual information to further improve the recognition accuracy. Moreover, the ability of deep generative models to effectively model data distribution has not been investigated yet in the field of sign language recognition. To this end, a novel approach for context-aware continuous sign language recognition using a generative adversarial network architecture, named as Sign Language Recognition Generative Adversarial Network (SLRGAN), is introduced. The proposed network architecture consists of a generator that recognizes sign language glosses by extracting spatial and temporal features from video sequences, as well as a discriminator that evaluates the quality of the generator’s predictions by modeling text information at the sentence and gloss levels. The paper also investigates the importance of contextual information on sign language conversations for both Deaf-to-Deaf and Deaf-to-hearing communication. Contextual information, in the form of hidden states extracted from the previous sentence, is fed into the bidirectional long short-term memory module of the generator to improve the recognition accuracy of the network. At the final stage, sign language translation is performed by a transformer network, which converts sign language glosses to natural language text. Our proposed method achieved word error rates of 23.4%, 2.1%, and 2.26% on the RWTH-Phoenix-Weather-2014 and the Chinese Sign Language (CSL) and Greek Sign Language (GSL) Signer Independent (SI) datasets, respectively.

show abstract

“…In recent years, feature fusion gradually receives attention. To make full use of different types of features, Su [37] combined the CNN and LSTM to form the fusion network, H. Zhou and W. Zhou also designed the spatial-temporal Multi-Cue network [38] for fully exploring the features from different cues.…”

Section: Network Learningmentioning

confidence: 99%

Attention-Based Sign Language Recognition Network Utilizing Keyframe Sampling and Skeletal Features

Pan

Zhang

2020

IEEE Access

View full text Add to dashboard Cite

Sign language recognition(SLR) is a multidisciplinary research topic in pattern recognition and computer vision. Due to large amount of data from the continuous frames of sign language videos, selecting representative data to eliminate irrelevant information has always been a challenging problem in data preprocessing of sign language samples. In recent years, skeletal data emerged as a new type of data but received insufficient attention. Meanwhile, due to the increasing diversity of sign language features, making full use of them has also been an important research topic. In this paper, we improve keyframecentered clips(KCC) sampling to get a new kind of sampling method called optimized keyframe-centered clips(OptimKCC) sampling to select key actions from sign language videos. Besides, we design a new kind of skeletal feature called Multi-Plane Vector Relation(MPVR) to describe the video samples. Finally, combined with the attention mechanism, we also use Attention-Based networks to distribute weights to the temporal features and the spatial features extracted from skeletal data. We implement comparison experiments on our own and the public sign language dataset under the Signer-Independent and the Signer-Dependent circumstances to show the advantages of our methods.

show abstract

Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition

Cited by 150 publications

References 55 publications

Better Sign Language Translation with STMC-Transformer

Better Sign Language Translation with STMC-Transformer

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network

Attention-Based Sign Language Recognition Network Utilizing Keyframe Sampling and Skeletal Features

Contact Info

Product

Resources

About