Sign Language Recognition, Generation, and Translation

Bragg, Danielle; Koller, Oscar; Bellard, Mary; Berke, Larwan; Boudreault, Patrick; Braffort, Annelies; Caselli, Naomi; Huenerfauth, Matt; Kacorri, Hernisa; Verhoef, Tessa; Vogler, Christian; Morris, Meredith Ringel

doi:10.1145/3308561.3353774

Cited by 282 publications

(158 citation statements)

References 78 publications

Supporting

Mentioning

154

Contrasting

Unclassified

Order By: Relevance

“…Hand posture recognition is a classical task in computer vision [1,6,8]. Despite many methods, which perform robustly under some limitations, the problem is still exciting.…”

Section: Resultsmentioning

confidence: 99%

“…The overall goal is to understand the body language and then create more functional and efficient human-computer interfaces. Application areas are vast; from driver support via hand-controlled cockpit elements [2], through to home automation with consumer electronics driven by gestures [3], gaming industry applications [4], interaction with virtual objects [5], and finally, technological support for people with disabilities [6]. Solutions available on the market are either limited, very simple, or have background and illumination requirements that are difficult to meet in real-life scenarios.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Hand Posture Recognition Using Skeletal Data and Distance Descriptor

Kapuściński

Warchoł

2020

Applied Sciences

View full text Add to dashboard Cite

In this paper, a method for the recognition of static hand postures based on skeletal data was presented. A novel descriptor was proposed. It encodes information about distances between particular hand points. Five different classifiers were tested, including four common methods and a proposed modification of nearest neighbor classifier, which can distinguish between posture classes differing mostly in hand orientation. The experiments were performed using three challenging datasets of gestures from Polish and American Sign Languages. The proposed method was compared with other approaches found in the literature. It outperforms every compared method, including our previous work, in terms of recognition rate.

show abstract

“…Hand posture recognition is a classical task in computer vision [1,6,8]. Despite many methods, which perform robustly under some limitations, the problem is still exciting.…”

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Hand Posture Recognition Using Skeletal Data and Distance Descriptor

Kapuściński

Warchoł

2020

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…It provides only 483 RGB samples in total. An extended list of sign language datasets can be found in [3], [42]. Montalbano Italian gesture dataset [43], which has recently become one of the most widely used isolated SLR datasets, contains 20 gestures and approximately 14,000 samples in total.…”

Section: A Sign Language Datasetsmentioning

confidence: 99%

AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods

2020

View full text Add to dashboard Cite

Sign language recognition is a challenging problem where signs are identified by simultaneous local and global articulations of multiple sources, i.e. hand shape and orientation, hand movements, body posture, and facial expressions. Solving this problem computationally for a large vocabulary of signs in real life settings is still a challenge, even with the state-of-the-art models. In this study, we present a new largescale multi-modal Turkish Sign Language dataset (AUTSL) with a benchmark and provide baseline models for performance evaluations. Our dataset consists of 226 signs performed by 43 different signers and 38,336 isolated sign video samples in total. Samples contain a wide variety of backgrounds recorded in indoor and outdoor environments. Moreover, spatial positions and the postures of signers also vary in the recordings. Each sample is recorded with Microsoft Kinect v2 and contains color image (RGB), depth, and skeleton modalities. We prepared benchmark training and test sets for user independent assessments of the models. We trained several deep learning based models and provide empirical evaluations using the benchmark; we used Convolutional Neural Networks (CNNs) to extract features, unidirectional and bidirectional Long Short-Term Memory (LSTM) models to characterize temporal information. We also incorporated feature pooling modules and temporal attention to our models to improve the performances. We evaluated our baseline models on AUTSL and Montalbano datasets. Our models achieved competitive results with the state-of-the-art methods on Montalbano dataset, i.e. 96.11% accuracy. In AUTSL random train-test splits, our models performed up to 95.95% accuracy. In the proposed user-independent benchmark dataset our best baseline model achieved 62.02% accuracy. The gaps in the performances of the same baseline models show the challenges inherent in our benchmark dataset. AUTSL benchmark dataset is publicly available at https://cvml.ankara.edu.tr.

show abstract

“…the VizWiz data challenge [11] calling for the computer vision community to work on visual question answering problems that can serve people with visual impairments. However, scarcity of large datasets generated from people with disabilities that can be used in AI-infused technologies remain one of the feld's biggest challenges [2]. While this is partly due to a smaller population [26], there are other factors specifc to these user groups.…”

mentioning

confidence: 99%

“…Here we adopt the following convention: Deaf (capitalized) describing members of the linguistic community of sign language users 2. Accessibility datasets here refer to data that can be used to train machine learning models and that are generated from people with disabilities and older adults.…”

mentioning

confidence: 99%

IncluSet: A Data Surfacing Repository for Accessibility Datasets

Kacorri

Dwivedi

Amancherla

et al. 2020

Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility

Self Cite

View full text Add to dashboard Cite

Datasets and data sharing play an important role for innovation, benchmarking, mitigating bias, and understanding the complexity of real world AI-infused applications. However, there is a scarcity of available data generated by people with disabilities with the potential for training or evaluating machine learning models. This is partially due to smaller populations, disparate characteristics, lack of expertise for data annotation, as well as privacy concerns. Even when data are collected and are publicly available, it is often difcult to locate them. We present a novel data surfacing repository, called IncluSet, that allows researchers and the disability community to discover and link accessibility datasets. The repository is pre-populated with information about 139 existing datasets: 65 made publicly available, 25 available upon request, and 49 not shared by the authors but described in their manuscripts. More importantly, IncluSet is designed to expose existing and new dataset contributions so they may be discoverable through Google Dataset Search.

show abstract

Sign Language Recognition, Generation, and Translation

Cited by 282 publications

References 78 publications

Hand Posture Recognition Using Skeletal Data and Distance Descriptor

Hand Posture Recognition Using Skeletal Data and Distance Descriptor

AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods

IncluSet: A Data Surfacing Repository for Accessibility Datasets

Contact Info

Product

Resources

About