Re-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs

Koller, Oscar; Zargaran, Sepehr; Ney, Hermann

doi:10.1109/cvpr.2017.364

Cited by 227 publications

(161 citation statements)

References 23 publications

Supporting

Mentioning

161

Contrasting

Order By: Relevance

“…The data collection method impacts both content and signer identity. For example, some corpora are formed of professional interpreters paid to interpret spoken content, such as news channels that provide interpreting [43,74,21]. Others are formed of expert signers paid to sign desired corpus content (e.g., [65,124,118]).…”

Section: Datasetsmentioning

confidence: 99%

Sign Language Recognition, Generation, and Translation

Bragg

Koller

Bellard

et al. 2019

The 21st International ACM SIGACCESS Conference on Computers and Accessibility

Self Cite

284

130

View full text Add to dashboard Cite

Developing successful sign language recognition, generation, and translation systems requires expertise in a wide range of fields, including computer vision, computer graphics, natural language processing, human-computer interaction, linguistics, and Deaf culture. Despite the need for deep interdisciplinary knowledge, existing research occurs in separate disciplinary silos, and tackles separate portions of the sign language processing pipeline. This leads to three key questions: 1) What does an interdisciplinary view of the current landscape reveal? 2) What are the biggest challenges facing the field? and 3) What are the calls to action for people working in the field? To help answer these questions, we brought together a diverse group of experts for a two-day workshop. This paper presents the results of that interdisciplinary workshop, providing key background that is often overlooked by computer scientists, a review of the state-of-the-art, a set of pressing challenges, and a call to action for the research community.Each group focused on the following questions:

show abstract

Section: Datasetsmentioning

confidence: 99%

Sign Language Recognition, Generation, and Translation

Bragg

Koller

Bellard

et al. 2019

The 21st International ACM SIGACCESS Conference on Computers and Accessibility

Self Cite

284

130

View full text Add to dashboard Cite

show abstract

“…a hidden Markov model (HMM), and the inter-class context, e.g. with a finite grammar [24,12,13]. While these approaches are particularly suited for videos that contain complex actions and have a Figure 1.…”

Section: Introductionmentioning

confidence: 99%

NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning

Richard

Kuehne

Iqbal

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

125

187

View full text Add to dashboard Cite

Video learning is an important task in computer vision and has experienced increasing interest over the recent years. Since even a small amount of videos easily comprises several million frames, methods that do not rely on a frame-level annotation are of special importance. In this work, we propose a novel learning algorithm with a Viterbibased loss that allows for online and incremental learning of weakly annotated video data. We moreover show that explicit context and length modeling leads to huge improvements in video segmentation and labeling tasks and include these models into our framework. On several action segmentation benchmarks, we obtain an improvement of up to 10% compared to current state-of-the-art methods.

show abstract

“…Then, the feature vector is constructed from the segmented signer and used as input to artificial neural network. An end-to-end sequence modelling using CNN-BiLSTM architecture usually used for gesture recognition was proposed for large vocabulary sign language recognition with RWTH-PHOENIX-Weather 2014 [32].…”

Section: Related Workmentioning

confidence: 99%

Neural Sign Language Translation Based on Human Keypoint Estimation

et al. 2019

View full text Add to dashboard Cite

We propose a sign language translation system based on human keypoint estimation. It is well-known that many problems in the field of computer vision require a massive amount of dataset to train deep neural network models. The situation is even worse when it comes to the sign language translation problem as it is far more difficult to collect high-quality training data. In this paper, we introduce the KETI (short for Korea Electronics Technology Institute) sign language dataset which consists of 14,672 videos of high resolution and quality. Considering the fact that each country has a different and unique sign language, the KETI sign language dataset can be the starting line for further research on the Korean sign language translation. Using the KETI sign language dataset, we develop a neural network model for translating sign videos into natural language sentences by utilizing the human keypoints extracted from a face, hands, and body parts. The obtained human keypoint vector is normalized by the mean and standard deviation of the keypoints and used as input to our translation model based on the sequence-to-sequence architecture. As a result, we show that our approach is robust even when the size of the training data is not sufficient. Our translation model achieves 93.28% (55.28%, respectively) translation accuracy on the validation set (test set, respectively) for 105 sentences that can be used in emergency situations. We compare several types of our neural sign translation models based on different attention mechanisms in terms of classical metrics for measuring the translation performance.

show abstract

Re-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs

Cited by 227 publications

References 23 publications

Sign Language Recognition, Generation, and Translation

Sign Language Recognition, Generation, and Translation

NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning

Neural Sign Language Translation Based on Human Keypoint Estimation

Contact Info

Product

Resources

About