Fingerspelling Recognition in the Wild With Iterative Visual Attention

Shi, Bowen; Rio, Aurora Martinez Del; Keane, Jonathan; Brentari, Diane; Shakhnarovich, Greg; Livescu, Karen

doi:10.1109/iccv.2019.00550

Cited by 47 publications

(64 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the basic status of convolutional neural networks (CNN) in deep learning networks, some research teams have conducted a series of CNN-based isolated sign language recognition studies since 2013 [ 6 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 ]. Based on CNN recognition, the algorithm can be optimized by adding multi-modal data (including depth, skeleton, key points of the human body, etc.…”

Section: Related Workmentioning

confidence: 99%

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Lü

2021

Sensors

View full text Add to dashboard Cite

Sign language is the most important way of communication for hearing-impaired people. Research on sign language recognition can help normal people understand sign language. We reviewed the classic methods of sign language recognition, and the recognition accuracy is not high enough because of redundant information, human finger occlusion, motion blurring, the diversified signing styles of different people, and so on. To overcome these shortcomings, we propose a multi-scale and dual sign language recognition Network (SLR-Net) based on a graph convolutional network (GCN). The original input data was RGB videos. We first extracted the skeleton data from them and then used the skeleton data for sign language recognition. SLR-Net is mainly composed of three sub-modules: multi-scale attention network (MSA), multi-scale spatiotemporal attention network (MSSTA) and attention enhanced temporal convolution network (ATCN). MSA allows the GCN to learn the dependencies between long-distance vertices; MSSTA can directly learn the spatiotemporal features; ATCN allows the GCN network to better learn the long temporal dependencies. The three different attention mechanisms, multi-scale attention mechanism, spatiotemporal attention mechanism, and temporal attention mechanism, are proposed to further improve the robustness and accuracy. Besides, a keyframe extraction algorithm is proposed, which can greatly improve efficiency by sacrificing a little accuracy. Experimental results showed that our method can reach 98.08% accuracy rate in the CSL-500 dataset with a 500-word vocabulary. Even on the challenging dataset DEVISIGN-L with a 2000-word vocabulary, it also reached a 64.57% accuracy rate, outperforming other state-of-the-art sign language recognition methods.

show abstract

Section: Related Workmentioning

confidence: 99%

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Lü

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Pre-Trained Models. The pre-trained models considered for this paper are all available for use with Keras 7 , which is the DL API that is used for this paper. All models are specialized in image classification, trained on the dataset ImageNet 8 .…”

Section: Transfer Learningmentioning

confidence: 99%

“…"This article is part of the topical collection "Recent Trends in Computer Vision" guest edited by P. Nagabhushan, Balasubramaniyan Raman, Satish Kumar Singh and Subrahmanyam Murala" Since the differences between signs in sign languages largely consist of different patterns like hand movements and shiftings, using artificial neural networks that learn from datasets of sign images is useful to develop a model able to detect and interpret signs [7].…”

Section: Introductionmentioning

confidence: 99%

Interpretation of Swedish Sign Language Using Convolutional Neural Networks and Transfer Learning

et al. 2021

View full text Add to dashboard Cite

The automatic interpretation of sign languages is a challenging task, as it requires the usage of high-level vision and high-level motion processing systems for providing accurate image perception. In this paper, we use Convolutional Neural Networks (CNNs) and transfer learning to make computers able to interpret signs of the Swedish Sign Language (SSL) hand alphabet. Our model consists of the implementation of a pre-trained InceptionV3 network, and the usage of the mini-batch gradient descent optimization algorithm. We rely on transfer learning during the pre-training of the model and its data. The final accuracy of the model, based on 8 study subjects and 9400 images, is 85%. Our results indicate that the usage of CNNs is a promising approach to interpret sign languages, and transfer learning can be used to achieve high testing accuracy despite using a small training dataset. Furthermore, we describe the implementation details of our model to interpret signs as a user-friendly web application.

show abstract

“…These models learn the relevant spatial or temporal parts of the image or video automatically from data. These models have also been used in the SLR domain [2], [8], [34], [36], [40].…”

Section: Related Workmentioning

confidence: 99%

“…In [40], ASL fingerspelling recognition model is proposed with iterative visual attention mechanism for real-life data. Fingerspelling is a part of sign language in which words are signed letter by letter.…”

Section: A Sign Language Datasetsmentioning

confidence: 99%

AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods

2020

View full text Add to dashboard Cite

Sign language recognition is a challenging problem where signs are identified by simultaneous local and global articulations of multiple sources, i.e. hand shape and orientation, hand movements, body posture, and facial expressions. Solving this problem computationally for a large vocabulary of signs in real life settings is still a challenge, even with the state-of-the-art models. In this study, we present a new largescale multi-modal Turkish Sign Language dataset (AUTSL) with a benchmark and provide baseline models for performance evaluations. Our dataset consists of 226 signs performed by 43 different signers and 38,336 isolated sign video samples in total. Samples contain a wide variety of backgrounds recorded in indoor and outdoor environments. Moreover, spatial positions and the postures of signers also vary in the recordings. Each sample is recorded with Microsoft Kinect v2 and contains color image (RGB), depth, and skeleton modalities. We prepared benchmark training and test sets for user independent assessments of the models. We trained several deep learning based models and provide empirical evaluations using the benchmark; we used Convolutional Neural Networks (CNNs) to extract features, unidirectional and bidirectional Long Short-Term Memory (LSTM) models to characterize temporal information. We also incorporated feature pooling modules and temporal attention to our models to improve the performances. We evaluated our baseline models on AUTSL and Montalbano datasets. Our models achieved competitive results with the state-of-the-art methods on Montalbano dataset, i.e. 96.11% accuracy. In AUTSL random train-test splits, our models performed up to 95.95% accuracy. In the proposed user-independent benchmark dataset our best baseline model achieved 62.02% accuracy. The gaps in the performances of the same baseline models show the challenges inherent in our benchmark dataset. AUTSL benchmark dataset is publicly available at https://cvml.ankara.edu.tr.

show abstract

Fingerspelling Recognition in the Wild With Iterative Visual Attention

Cited by 47 publications

References 33 publications

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Interpretation of Swedish Sign Language Using Convolutional Neural Networks and Transfer Learning

AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods

Contact Info

Product

Resources

About