Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and hearing people

Qu, Xiao; Qin, Minying; Yin, Yuting

doi:10.1016/j.neunet.2020.01.030

Cited by 83 publications

(46 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, deep learning approaches have been applied to the task of SLP [15,53,61]. Stoll et al present an initial SLP model using a combination of Neural Machine Translation (NMT) and Generative Adversarial Networks (GANs) [54].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Progressive Transformers for End-to-End Sign Language Production

Saunders

Camgöz

Bowden

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The goal of automatic Sign Language Production (SLP) is to translate spoken language to a continuous stream of sign language video at a level comparable to a human translator. If this was achievable, then it would revolutionise Deaf hearing communications. Previous work on predominantly isolated SLP has shown the need for architectures that are better suited to the continuous domain of full sign sequences. In this paper, we propose Progressive Transformers, the first SLP model to translate from discrete spoken language sentences to continuous 3D sign pose sequences in an end-to-end manner. A novel counter decoding technique is introduced, that enables continuous sequence generation at training and inference. We present two model configurations, an end-toend network that produces sign direct from text and a stacked network that utilises a gloss intermediary. We also provide several data augmentation processes to overcome the problem of drift and drastically improve the performance of SLP models. We propose a back translation evaluation mechanism for SLP, presenting benchmark quantitative results on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset and setting baselines for future research. Code available at https://github.com/BenSaunders27/ ProgressiveTransformersSLP.

show abstract

Section: Related Workmentioning

confidence: 99%

“…We liken it to the wide use of the inception score for generative models [51], using a pre-trained classifier. Similarly, recent SLP work used an SLR discriminator to evaluate isolated skeletons [61], but did not measure the translation performance.…”

Section: Evaluation Detailsmentioning

confidence: 99%

Progressive Transformers for End-to-End Sign Language Production

Saunders

Camgöz

Bowden

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…For example, a leap motion sensor was used with a hidden Markov layer for pattern recognition of 24 gestures of ASL [41] and instrumented gloves with flex [42] and/or inertial sensors [43] with machine learning techniques reached accuracies ranging from 86% to 98%. A combination of images and deep learning techniques demonstrated high performance for this task, such as for Persian Sign Language [44] and CSL [45]. This indicates that this topic has been developed recently.…”

Section: Related Workmentioning

confidence: 82%

Analysis of Influence of Segmentation, Features, and Classification in sEMG Processing: A Case Study of Recognition of Brazilian Sign Language Alphabet

Mendes

Freitas

Campos

et al. 2020

Sensors

View full text Add to dashboard Cite

Sign Language recognition systems aid communication among deaf people, hearing impaired people, and speakers. One of the types of signals that has seen increased studies and that can be used as input for these systems is surface electromyography (sEMG). This work presents the recognition of a set of alphabet gestures from Brazilian Sign Language (Libras) using sEMG acquired from an armband. Only sEMG signals were used as input. Signals from 12 subjects were acquired using a MyoTM armband for the 26 signs of the Libras alphabet. Additionally, as the sEMG has several signal processing parameters, the influence of segmentation, feature extraction, and classification was considered at each step of the pattern recognition. In segmentation, window length and the presence of four levels of overlap rates were analyzed, as well as the contribution of each feature, the literature feature sets, and new feature sets proposed for different classifiers. We found that the overlap rate had a high influence on this task. Accuracies in the order of 99% were achieved for the following factors: segments of 1.75 s with a 12.5% overlap rate; the proposed set of four features; and random forest (RF) classifiers.

show abstract

“…When the hexapod robot is half occupied, it moves forward in the tripodal gait [17]. In this case, the robot legs take the same amount of time in swing phase as in support phase.…”

Section: Phase Relationship Of Each Legmentioning

confidence: 99%

Gait Control of Hexapod Robot Based on Field-Programmable Gate Array and Central Pattern Generator

Qiao¹

2020

JESA

View full text Add to dashboard Cite

This paper attempts to improve the terrain adaptability of hexapod robot through gait control. Firstly, the multi-leg coupling in the tripodal gait of the hexapod robot was modeled by Hopf oscillator. Then, annular central pattern generator (CPG) was adopted to simulate the leg movements of hexapod robot between signals. Furthermore, a physical prototype was designed for the gait control test on field-programmable gate array (FPGA), and the algorithm of the rhythmic output of the model was programmed in Verilog, a hardware description language. Finally, the effectiveness of our gait control method was verified through the simulation on Xilinx. The results show that the phase difference of the CPG network remained stable; the designed hexapod robot moved at about 5.15cm/s stably in a tripodal gait, and outperformed wheeled and tracked robots in terrain adaptation. The research findings lay a solid basis for the design of all-terrain multi-leg robots.

show abstract

Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and hearing people

Cited by 83 publications

References 15 publications

Progressive Transformers for End-to-End Sign Language Production

Progressive Transformers for End-to-End Sign Language Production

Analysis of Influence of Segmentation, Features, and Classification in sEMG Processing: A Case Study of Recognition of Brazilian Sign Language Alphabet

Gait Control of Hexapod Robot Based on Field-Programmable Gate Array and Central Pattern Generator

Contact Info

Product

Resources

About