A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion

Mittal, Anshul; Kumar, Pradeep; Roy, Partha Pratim; Raman, Balasubramanian; Chaudhuri, B. B.

doi:10.1109/jsen.2019.2909837

Cited by 189 publications

(67 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, the BRNN model outputs the recognition result. recognition system for continuous sign language by using LMC [25]. In the system, the authors presented a modified Long Short-Term Memory model which has three input gates and an output gate, adding a 2D-Convolutional Neural Network to the input gate that receives feature inputs.…”

Section: Methodsmentioning

confidence: 99%

“…Finally, by using the leave-one-person-out, 2-fold, and 10-fold cross-validation, the recognition accuracy results for 0-9 are 94.2%, 95.1%, and 90.2%, respectively, and for A-Z are 89.2%, 92.9%, and 86.4%, respectively. Mittal et al proposed a recognition system for continuous sign language by using LMC [25]. In the system, the authors presented a modified Long Short-Term Memory model which has three input gates and an output gate, adding a 2D-Convolutional Neural Network to the input gate that receives feature inputs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Hand Gesture Recognition Based on a Leap Motion Controller and Two-Layer Bidirectional Recurrent Neural Network

Yang

Chen

Zhu

2020

Sensors

View full text Add to dashboard Cite

Dynamic hand gesture recognition is one of the most significant tools for human–computer interaction. In order to improve the accuracy of the dynamic hand gesture recognition, in this paper, a two-layer Bidirectional Recurrent Neural Network for the recognition of dynamic hand gestures from a Leap Motion Controller (LMC) is proposed. In addition, based on LMC, an efficient way to capture the dynamic hand gestures is identified. Dynamic hand gestures are represented by sets of feature vectors from the LMC. The proposed system has been tested on the American Sign Language (ASL) datasets with 360 samples and 480 samples, and the Handicraft-Gesture dataset, respectively. On the ASL dataset with 360 samples, the system achieves accuracies of 100% and 96.3% on the training and testing sets. On the ASL dataset with 480 samples, the system achieves accuracies of 100% and 95.2%. On the Handicraft-Gesture dataset, the system achieves accuracies of 100% and 96.7%. In addition, 5-fold, 10-fold, and Leave-One-Out cross-validation are performed on these datasets. The accuracies are 93.33%, 94.1%, and 98.33% (360 samples), 93.75%, 93.5%, and 98.13% (480 samples), and 88.66%, 90%, and 92% on ASL and Handicraft-Gesture datasets, respectively. The developed system demonstrates similar or better performance compared to other approaches in the literature.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Dynamic Hand Gesture Recognition Based on a Leap Motion Controller and Two-Layer Bidirectional Recurrent Neural Network

Yang

Chen

Zhu

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…A combination of bi-directional LSTM, CNN, and CRF was developed for sequence labeling in [54]. Bidirectional LSTM has shown a successful performance in action recognition [55] and sign language recognition [19], [56], [57].…”

Section: Related Workmentioning

confidence: 99%

DeepArSLR: A Novel Signer-Independent Deep Learning Framework for Isolated Arabic Sign Language Gestures Recognition

Aly

2020

IEEE Access

110

View full text Add to dashboard Cite

Hand gesture recognition has attracted the attention of many researchers due to its wide applications in robotics, games, virtual reality, sign language and human-computer interaction. Sign language is a structured form of hand gestures and the most effective communication way among hear-impaired people. Developing an efficient sign language recognition system to recognize dynamic isolated gestures encounters three major challenges, namely, hand segmentation, hand shape feature representation and gesture sequence recognition. Traditional sign language recognition methods utilize color-based hand segmentation algorithms to segment hands, hand-crafted feature extraction for hand shape representation and Hidden Markov Model (HMM) for sequence recognition. In this paper, a novel framework is proposed for signerindependent sign language recognition using multiple deep learning architectures comprising hand semantic segmentation, hand shape feature representation and deep recurrent neural network. The recently developed semantic segmentation method called DeepLabv3+ is trained using a set of pixel-labeled hand images to extract hand regions from each frame of the input video. Then, the extracted hand regions are cropped and scaled to a fixed size to alleviate hand scale variations. Extracting hand shape features is achieved using a single layer Convolutional Self-Organizing Map (CSOM) instead of relying on transfer learning of pretrained deep convolutional neural networks. The sequence of extracted feature vectors are then recognized using deep Bi-directional Long Short-Term Memory (BiLSTM) recurrent neural network. BiLSTM network contains three BiLSTM layers, one fully connected and softmax layers. The performance of the proposed method is evaluated using a challenging Arabic sign language database containing 23 isolated words captured from three different users. Experimental results show that the performance of proposed framework outperforms with large margin the state-of-the-art methods for signer-independent testing strategy. INDEX TERMS Arabic sign language recognition, deep learning, hand semantic segmentation, convolutional self-organizing map, signer-independent, deep BiLSTM network.

show abstract

“…In addition to the correlation network, a bidirectional LSTM network is also used in the model for learning the temporal dynamics in the data. CNN-LSTM combined network was initially introduced for sequence prediction applications with spatial inputs like images or videos [13]. Interestingly, a few recent studies have adopted CNN-LSTM networks for analyzing 1-D signals.…”

Section: Introductionmentioning

confidence: 99%

Time Series Classification-Based Correlational Neural Network With Bidirectional LSTM for Automated Detection of Kidney Disease

2021

View full text Add to dashboard Cite

In this paper, we aim to explore the feasibility of salivary analysis for Chronic Kidney Disease (CKD) detection and thereby design an automated mechanism to detect CKD through analysis of human saliva samples. We have implemented an improved deep learning model that combines both a one-dimensional Correlational Neural Network (1-D CorrNN) and bidirectional Long Short-Term Memory (LSTM) network for making accurate predictions. The LSTM network is integrated with the neural model to utilize the capabilities of both these networks to analyze the time-series data. The proposed model is trained and tested with a CKD sensing module. The application of deep learning algorithms helps to improve the detection accuracy as they are capable of discovering the best features from the input data. The proposed method achieved an average accuracy rate of 98.08% for the testing dataset. The results show that the proposed detection module and classification algorithm substantially advance the current methodologies, and provides more accurate predictions compared to conventional methods.

show abstract

A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion

Cited by 189 publications

References 24 publications

Dynamic Hand Gesture Recognition Based on a Leap Motion Controller and Two-Layer Bidirectional Recurrent Neural Network

Dynamic Hand Gesture Recognition Based on a Leap Motion Controller and Two-Layer Bidirectional Recurrent Neural Network

DeepArSLR: A Novel Signer-Independent Deep Learning Framework for Isolated Arabic Sign Language Gestures Recognition

Time Series Classification-Based Correlational Neural Network With Bidirectional LSTM for Automated Detection of Kidney Disease

Contact Info

Product

Resources

About