Data-driven development of Virtual Sign Language Communication Agents

Balayn, Agathe; Brock, Heike; Nakadai, Kazuhiro

doi:10.1109/roman.2018.8525717

Cited by 5 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This observation should be very interesting and valuable in the future and help to reduce the overall number of necessary sign tracking features. Moreover, our results considerably outperform results reported for an end-to-end approach utilizing a reduced number of sentence collections of the same data set [ 31 ]. As such, the staged approach appears meaningful for small, sparse and insufficient data sets with high NME content, which are otherwise hard to learn.…”

Section: Discussionmentioning

confidence: 44%

“…Previous efforts to implement a bidirectional sequence-to-sequence system from joint position data [ 31 ] show that it is very difficult to learn a working system for the given data set due to its high variability induced by the presence of linguistic features. For this reason, we perform our following analysis under a two-stage system with temporal segmentation.…”

Section: Cslr System Pipelinementioning

confidence: 99%

See 1 more Smart Citation

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Brock

Farag

Nakadai

2020

Sensors

Self Cite

View full text Add to dashboard Cite

The quality of recognition systems for continuous utterances in signed languages could be largely advanced within the last years. However, research efforts often do not address specific linguistic features of signed languages, as e.g., non-manual expressions. In this work, we evaluate the potential of a single video camera-based recognition system with respect to the latter. For this, we introduce a two-stage pipeline based on two-dimensional body joint positions extracted from RGB camera data. The system first separates the data flow of a signed expression into meaningful word segments on the base of a frame-wise binary Random Forest. Next, every segment is transformed into image-like shape and classified with a Convolutional Neural Network. The proposed system is then evaluated on a data set of continuous sentence expressions in Japanese Sign Language with a variation of non-manual expressions. Exploring multiple variations of data representations and network parameters, we are able to distinguish word segments of specific non-manual intonations with 86% accuracy from the underlying body joint movement data. Full sentence predictions achieve a total Word Error Rate of 15.75%. This marks an improvement of 13.22% as compared to ground truth predictions obtained from labeling insensitive towards non-manual content. Consequently, our analysis constitutes an important contribution for a better understanding of mixed manual and non-manual content in signed communication.

show abstract

Section: Discussionmentioning

confidence: 44%

Section: Cslr System Pipelinementioning

confidence: 99%

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Brock

Farag

Nakadai

2020

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…A common approach to sign language representation is the use of 3D avatars that with a high degree of accuracy and realism can reproduce facial expressions and body/hand movements in a way that represent signs understandable by deaf or hearing-impaired people. Balayn et al in [ 99 ], developed a virtual communication agent for sign language to recognize Japanese sign language sentences from video recordings and synthesize sign language animations. Their system adopted a deep LSTM encoder-decoder network to translate sign language videos to spoken text, while a separate encoder-decoder network used as input the sign language glosses and extracted specific encodings, which were then used to synthesize the avatar motion.…”

Section: Sign Language Representationmentioning

confidence: 99%

Artificial Intelligence Technologies for Sign Language

Papastratis

Chatzikonstantinou

Konstantinidis

et al. 2021

Sensors

View full text Add to dashboard Cite

AI technologies can play an important role in breaking down the communication barriers of deaf or hearing-impaired people with other communities, contributing significantly to their social inclusion. Recent advances in both sensing technologies and AI algorithms have paved the way for the development of various applications aiming at fulfilling the needs of deaf and hearing-impaired communities. To this end, this survey aims to provide a comprehensive review of state-of-the-art methods in sign language capturing, recognition, translation and representation, pinpointing their advantages and limitations. In addition, the survey presents a number of applications, while it discusses the main challenges in the field of sign language technologies. Future research direction are also proposed in order to assist prospective researchers towards further advancing the field.

show abstract

“…Other examples of normalization can be observed in the experiments by [67], [70], [71].Balayn et al [67] normalized Japanese sign language (JSL) motion sentences and used them as inputs and outputs for Seq2Seq models. Konstantinidis et al [70] normalized hand positions, which were used as inputs for the classifier together with cropped hand regions.…”

Section: ) Normalization and Filteringmentioning

confidence: 99%

“…Koller et al [63] utilized 1024-dimensional feature maps and applied PCA to reduce the dimensionality to 200. Another experiment by [67] used PCA to select data streams exhibiting a high variance represented by approximately 492 dimensions. The use of PCA on Kinect data has also helped to reduce cases of overfitting.…”

Section: Principal Component Analysismentioning

confidence: 99%

Deep Learning for Sign Language Recognition: Current Techniques, Benchmarks, and Open Issues

Al‐Qurishi¹,

Khalid²,

Souissi³

2021

IEEE Access

View full text Add to dashboard Cite

People with hearing impairments are found worldwide; therefore, the development of effective local level sign language recognition (SLR) tools is essential. We conducted a comprehensive review of automated sign language recognition based on machine/deep learning methods and techniques published between 2014 and 2021 and concluded that the current methods require conceptual classification to interpret all available data correctly. Thus, we turned our attention to elements that are common to almost all sign language recognition methodologies. This paper discusses their relative strengths and weaknesses, and we propose a general framework for researchers. This study also indicates that input modalities bear great significance in this field; it appears that recognition based on a combination of data sources, including vision-based and sensor-based channels, is superior to a unimodal analysis. In addition, recent advances have allowed researchers to move from simple recognition of sign language characters and words towards the capacity to translate continuous sign language communication with minimal delay. Many of the presented models are relatively effective for a range of tasks, but none currently possess the necessary generalization potential for commercial deployment. However, the pace of research is encouraging, and further progress is expected if specific difficulties are resolved.

show abstract

Data-driven development of Virtual Sign Language Communication Agents

Cited by 5 publications

References 19 publications

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Recognition of Non-Manual Content in Continuous Japanese Sign Language

Artificial Intelligence Technologies for Sign Language

Deep Learning for Sign Language Recognition: Current Techniques, Benchmarks, and Open Issues

Contact Info

Product

Resources

About