Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production

Abstract

Recent approaches to Sign Language Production (SLP) have adopted spoken language Neural Machine Translation (NMT) architectures, applied without sign-specific modifications. In addition, these works represent sign language as a sequence of skeleton pose vectors, projected to an abstract representation with no inherent skeletal structure.In this paper, we represent sign language sequences as a skeletal graph structure, with joints as nodes and both spatial and temporal connections as edges. To operate on this graphical structure, we propose Skeletal Graph Self-Attention (SGSA), a novel graphical attention layer that embeds a skeleton inductive bias into the SLP model. Retaining the skeletal feature representation throughout, we directly apply a spatio-temporal adjacency matrix into the self-attention formulation. This provides structure and context to each skeletal joint that is not possible when using a non-graphical abstract representation, enabling fluid and expressive sign language production. We evaluate our Skeletal Graph Self-Attention architecture on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset, achieving state-of-the-art back translation performance with an 8% and 7% improvement over competing methods for the dev and test sets.

Synthetic Corpus Generation for Deep Learning-Based Translation of Spanish Sign Language

“…Furthermore, there are recent studies [ 29 , 30 ] that try to solve the problem inversely, i.e., instead of providing hearing–deaf communication, they focus on creating a system in which a hearing person can be understood by any deaf person through the generation of videos or a sequence of still images, also known as Sign Language Production (SLP).…”

“…In the example, in Spanish, the translated sentence obtained from SLT is "I have an appointment to get my university degree" which corresponds to gloss annotation, obtained from CSLR, "Me appointment reason/why degree my university get". Furthermore, there are recent studies [29,30] that try to solve the problem inversely, i.e., instead of providing hearing-deaf communication, they focus on creating a system in which a hearing person can be understood by any deaf person through the generation of videos or a sequence of still images, also known as Sign Language Production (SLP).…”