Cross-modal Neural Sign Language Translation

Abstract

Sign Language is the primary means of communication for the majority of the Deaf and hard-of-hearing communities. Current computational approaches in this general research area have focused specifically on sign language recognition and the translation of sign language to text. However, the reverse problem of translating from spoken to sign language has so far not been widely explored. The goal of this doctoral research is to explore sign language translation in this generalized setting, i.e. translating from spoken language to sign language and vice versa. Towards that end, we propose a concrete methodology for tackling the problem of speech to sign language translation and introduce How2Sign, the first public, continuous American Sign Language dataset that enables such research. With a parallel corpus of almost 60 hours of sign language videos (collected with both RGB and depth sensor data) and the corresponding speech transcripts for over 2500 instructional videos, How2Sign is a public dataset of unprecedented scale that can be used to advance not only sign language translation, but also a wide range of sign language understanding tasks.

Progressive Transformers for End-to-End Sign Language Production

“…Distinct to SLR, the task of SLT was recently introduced by Camgoz et al [7], aiming to directly translate sign videos to spoken language sentences [15,31,45,62]. SLT is more challenging than CSLR due to the differences in grammar and ordering between sign and spoken language.…”

“…Recently, deep learning approaches have been applied to the task of SLP [15,53,61]. Stoll et al present an initial SLP model using a combination of Neural Machine Translation (NMT) and Generative Adversarial Networks (GANs) [54].…”

Two Dimensional Sign Language Agent

“…For the model, we created a simple 2D human (see figure 1a) in Inkscape 3 . We used a 2D model instead of a 3D model because OpenPose can only estimate a 2D pose from a single camera perspective.…”

“…An early example is TESSA [2], which translated from spoken language to British Sign Language in the postoffice domain. More recent approaches have adopted sequenceto-sequence approaches to translate a sequence of text into the corresponding signs [3,5]. However, these examples of automatic sign language generation all suffer from the use of technologies that are not highly available (motion capture, depth cameras, and large computational power respectively) leading to them being less general, portable, scalable, and ultimately usable.…”

Deep Learning for Sign Language Recognition: Current Techniques, Benchmarks, and Open Issues

“…SLR is a field dedicated to the automated interpretation of hand gestures and other signs used in communications between people with a speech or hearing impairment. Because hardware and software components have evolved to the point where developing advanced systems with real-time translation capacities appear to be within reach, a large number of exciting and innovative solutions have been proposed and tested in recent years [5]- [9]with the objective of building fully functional systems that can understand sign language and respond to commands given in this format. However, before any truly practical applications can be considered, it is imperative to perfect the interpretation algorithms to the point where false positives are rare [6], [10]- [13].…”