Towards Fast and High-Quality Sign Language Production

Abstract

Sign Language Production (SLP) aims to automatically translate a spoken language description to its corresponding sign language video. The core procedure of SLP is to transform sign gloss intermediaries into sign pose sequences (G2P). Most existing methods for G2P are based on sequential autoregression or sequence-tosequence encoder-decoder learning. However, by generating target pose frames conditioned on the previously generated ones, these models are prone to bringing issues such as error accumulation and high inference latency. In this paper, we argue that such issues are mainly caused by adopting autoregressive manner. Hence, we propose a novel Non-AuToregressive (NAT) model with a parallel decoding scheme, as well as an External Aligner for sequence alignment learning. Specifically, we extract alignments from the external aligner by monotonic alignment search for gloss duration prediction, which is used by a length regulator to expand the source gloss sequence to match the length of the target sign pose sequence for parallel sign pose generation. Furthermore, we devise a spatialtemporal graph convolutional pose generator in the NAT model to generate smoother and more natural sign pose sequences. Extensive experiments conducted on PHOENIX14T dataset show that our proposed model outperforms state-of-the-art autoregressive models in terms of speed and quality. CCS CONCEPTS• Computing methodologies → Motion capture; Activity recognition and understanding.

Modeling Intensification for Sign Language Generation: A Computational Approach

“…It is worth noting that, as stated by (Huang et al, 2021), the proposed decoding mechanism provides weak supervisions with the initial ground-truth frame and guided counter sequences during the inference time.…”

Spatial–Temporal Graph Transformer With Sign Mesh Regression for Skinned-Based Sign Language Production

“…Further, Saunders et al [6] proposed a spatial-temporal skeletal graph attention layer that embeds a hierarchical body inductive bias into the self-attention mechanism. Huang et al [4] developed spatial-temporal graph convolution layers into the pose generator which is able to capture both intraframe and inter-frame information of sign language videos. However, all these methods disregard each joint has different contributions to gestures expression.…”

“…Recently, Transformer-based methods [1], [2], [3], [4], [5] became the most widespread methods to produce skeletons for SLP. However, there is still a problem in these works: such architecture always ignores the structural relationships of the human skeletons, by which poor performance would be obtained.…”

“…However, there is still a problem in these works: such architecture always ignores the structural relationships of the human skeletons, by which poor performance would be obtained. Thereupon, the existing SLP method [4] devises a spatial-temporal graph convolution (GCN) as pose generator which implemented from a standard 2D convolution. Skeletal graph self-attention [6] encodes the spatio-temporal connectivity into the node features while calculating attention matrices.…”

Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production

“…Recently, there have been many deep learning approaches to SLP proposed [23,42,48,50,52,56,63,71], with Saunders et al achieving state-of-the-art results with gloss supervision [52]. These works predominantly represent sign languages as sequences of skeletal frames, with each frame encoded as a vector of joint coordinates [51] that disregards any spatio-temporal structure available within a skeletal representation.…”