Progressive Transformers for End-to-End Sign Language Production

Abstract

The goal of automatic Sign Language Production (SLP) is to translate spoken language to a continuous stream of sign language video at a level comparable to a human translator. If this was achievable, then it would revolutionise Deaf hearing communications. Previous work on predominantly isolated SLP has shown the need for architectures that are better suited to the continuous domain of full sign sequences. In this paper, we propose Progressive Transformers, the first SLP model to translate from discrete spoken language sentences to continuous 3D sign pose sequences in an end-to-end manner. A novel counter decoding technique is introduced, that enables continuous sequence generation at training and inference. We present two model configurations, an end-toend network that produces sign direct from text and a stacked network that utilises a gloss intermediary. We also provide several data augmentation processes to overcome the problem of drift and drastically improve the performance of SLP models. We propose a back translation evaluation mechanism for SLP, presenting benchmark quantitative results on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset and setting baselines for future research. Code available at https://github.com/BenSaunders27/ ProgressiveTransformersSLP.

Artificial Intelligence Technologies for Sign Language

“…This approach resulted in smooth transitions between glosses and refined details on hand and finger shapes. Saunders et al in [ 114 ], employed Transformers to automatically generate 3D human poses from spoken text using a multiple-level configuration. A text-to-gloss-to-pose (T2G2P) network with Transformer layers translated text sentences to sign language glosses and finally to 3D poses, while a text-to-pose (T2P) network directly transformed text into human poses.…”

Including Signed Languages in Natural Language Processing

“…On RWTH-PHOENIX-Weather 2014T, we obtain 22.17 BLEU on testing; on Public DGS Corpus, we obtain a mere 3.2 BLEU. Although Transformers achieve encouraging results on RWTH-PHOENIX-Weather 2014T (Saunders et al, 2020b;, they fail on more realistic, opendomain data. These results reveal that firstly, for real-world applications, we need more data to train such types of models, and secondly, while available data is severely limited in size, less data-hungry and more linguistically-informed approaches may be more suitable.…”

Towards Fast and High-Quality Sign Language Production

“…Sign language, a rich visual language with complex grammatical structures, is the language of communication for the Deaf community. To involve the Deaf in the predominantly spoken language of the wider world, a large amount of methods [38][39][40]43] have been recently proposed to tackle the challenging Sign Language Production (SLP) problem. Given a spoken language description, SLP aims to automatically translate it into the corresponding continuous sign sequence.…”

“…Given a spoken language description, SLP aims to automatically translate it into the corresponding continuous sign sequence. Generally, sign sequences can be represented as sign skeleton pose sequences [38,40] or sign language videos [39].…”

“…As shown in Figure 1, previous works for SLP usually translate the spoken language into gloss 1 intermediary first, then generate sign pose sequence from the gloss sequence (G2P) [38,40], and finally maybe use the sign pose sequence to drive a style image to generate a sign language video [39]. Thereinto, G2P, the core procedure of this pipeline, is the focus of this paper.…”