3D Avatar Approach for Continuous Sign Movement Using Speech/Text

Abstract

Sign language is a visual language for communication used by hearing-impaired people with the help of hand and finger movements. Indian Sign Language (ISL) is a well-developed and standard way of communication for hearing-impaired people living in India. However, other people who use spoken language always face difficulty while communicating with a hearing-impaired person due to lack of sign language knowledge. In this study, we have developed a 3D avatar-based sign language learning system that converts the input speech/text into corresponding sign movements for ISL. The system consists of three modules. Initially, the input speech is converted into an English sentence. Then, that English sentence is converted into the corresponding ISL sentence using the Natural Language Processing (NLP) technique. Finally, the motion of the 3D avatar is defined based on the ISL sentence. The translation module achieves a 10.50 SER (Sign Error Rate) score.

Evolution and Trends in Sign Language Avatar Systems: Unveiling a 40-Year Journey via Systematic Review

“…Interact. 2023, 7, x FOR PEER REVIEW 15 of 36 [76] 3d avatar approach for continuous sign movement using speech/text 2021 journal 3 [77] Automatic generation of a 3D sign language avatar on AR glasses given 2D videos of human signers 2021 conference 1 [78] SignPose: Sign Language Animation Through 3D Pose Lifting 2021 conference 1 [79] An automatic machine translation system for multi-lingual speech to Indian sign language 2022 journal 1 [80] Holographic sign language avatar interpreter: A user interaction study in a mixed reality classroom 2022 journal 1…”

“…Sign Synthesis Techniques [76] The methods used for SL synthesis involve a combination of speech recognition and script generation techniques. The "IBM-Watson speech to text" service [87] is utilized to convert English audio recordings into text.…”

“…[61] used 3D Studio Max [91] for character modeling. Some papers also used the Blender software [92] for character modeling, rigging, and movement such as [60,66,76]. Ref.…”

Artificial Intelligence Technologies for Sign Language

“…Das et al in [103], developed a 3D avatar to convert Indian text or speech into sign language. The input was translated to English and then to the corresponding Indian sign language using Natural Language Processing (NLP) rules and techniques.…”

Effects of Sinusoidal Model on Non-Parallel Voice Conversion with Adversarial Learning

“…This is achieved through adjusting the spectral and prosodic features of the input speaker [1]. This technology has been applied to many potential tasks, such as speech synthesis [2], speech enhancement [3,4], normalization of impaired speech [5,6], and singing style conversion [7] and can also be used for generating new voices for animated and fictional movies. Towards the practical use of these applications, it is necessary to modify the VC approach.…”