Sign Language Avatars: A Question of Representation

Abstract

Given the achievements in automatically translating text from one language to another, one would expect to see similar advancements in translating between signed and spoken languages. However, progress in this effort has lagged in comparison. Typically, machine translation consists of processing text from one language to produce text in another. Because signed languages have no generally-accepted written form, translating spoken to signed language requires the additional step of displaying the language visually as animation through the use of a three-dimensional (3D) virtual human commonly known as an avatar. Researchers have been grappling with this problem for over twenty years, and it is still an open question. With the goal of developing a deeper understanding of the challenges posed by this question, this article gives a summary overview of the unique aspects of signed languages, briefly surveys the technology underlying avatars and performs an in-depth analysis of the features in a textual representation for avatar display. It concludes with a comparison of these features and makes observations about future research directions.

Special issue on sign language translation and avatar technology

“…How best to represent sign language for machine translation is still an open question. Some approaches to representing sign language, such as motion capture, contain so much detail that it is prohibitive to use them to generate new signed utterances [6]. At the other extreme, there are representations, such as glosses that are so parsimonious that they are missing data essential for effective language sign generation.…”

Trends and Challenges for Sign Language Recognition with Machine Learning

“…Instead, it is more ethical to communicate with SL users and ask them what technologies they want or not. For instance, there is some resistance against SL avatars [2,3,4]. Recent research projects such as SignON [5] and EASIER [6] aim to enable communication between researchers in AI, who are often hearing, SL linguists, and the end users of potential SL applications.…”

JUMLA-QSL-22: A Novel Qatari Sign Language Continuous Dataset

“…Compounding this challenge is the complexity of sign language representation. The representation must encompass essential linguistic information, movements, speed, and other variables to ensure no loss of semantic information [10]. Even slight changes in timing, movement, or configuration of different gesture components could lead to entirely different meanings [11].…”