Bengali Sign Language Recognition Using Deep Convolutional Neural Network

Hossen, Mohammad Anwar; Govindaiah, Arun; Sultana, Sadia; Bhuiyan, Alauddin

doi:10.1109/iciev.2018.8640962

Cited by 53 publications

(24 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…), detecting hand regions, and feature fusion. Literature [ 26 , 27 , 28 , 29 ] proposed a sign language recognition CNN network based on multi-modal data, which can use multi-scale to capture image features at various levels. Kopuklu et al [ 28 ] proposed a data-level fusion strategy for fusing motion information into static images, and sent the fused spatiotemporal features to the CNN network for subsequent classification, and achieved commendable recognition.…”

Section: Related Workmentioning

confidence: 99%

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Lü

2021

Sensors

View full text Add to dashboard Cite

Sign language is the most important way of communication for hearing-impaired people. Research on sign language recognition can help normal people understand sign language. We reviewed the classic methods of sign language recognition, and the recognition accuracy is not high enough because of redundant information, human finger occlusion, motion blurring, the diversified signing styles of different people, and so on. To overcome these shortcomings, we propose a multi-scale and dual sign language recognition Network (SLR-Net) based on a graph convolutional network (GCN). The original input data was RGB videos. We first extracted the skeleton data from them and then used the skeleton data for sign language recognition. SLR-Net is mainly composed of three sub-modules: multi-scale attention network (MSA), multi-scale spatiotemporal attention network (MSSTA) and attention enhanced temporal convolution network (ATCN). MSA allows the GCN to learn the dependencies between long-distance vertices; MSSTA can directly learn the spatiotemporal features; ATCN allows the GCN network to better learn the long temporal dependencies. The three different attention mechanisms, multi-scale attention mechanism, spatiotemporal attention mechanism, and temporal attention mechanism, are proposed to further improve the robustness and accuracy. Besides, a keyframe extraction algorithm is proposed, which can greatly improve efficiency by sacrificing a little accuracy. Experimental results showed that our method can reach 98.08% accuracy rate in the CSL-500 dataset with a 500-word vocabulary. Even on the challenging dataset DEVISIGN-L with a 2000-word vocabulary, it also reached a 64.57% accuracy rate, outperforming other state-of-the-art sign language recognition methods.

show abstract

Section: Related Workmentioning

confidence: 99%

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Lü

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Many researchers applied CNN to recognize BdSL. Islam et al [19], Hoque et al [20], and Islam et al [21] utilized CNN to recognize two-handed BdSL alphabets, whereas, Rony et al [22], Hossen et al [23], and Rafi et al [24] implemented CNN to recognize one-handed…”

Section: Literature Reviewmentioning

confidence: 99%

Design and Development of a Humanoid Robot for Sign Language Interpretation

et al. 2021

View full text Add to dashboard Cite

Purpose With an indisputable complexity of communication for hearing and speaking impaired people, most sign language recognition systems utilize virtual reality or onscreen robots. This paper presents the design and development of a special and low-cost humanoid robot that can perform as a sign language interpreter. To the best of our knowledge, this is the first endeavor to fabricate a humanoid robot for Bangla sign language (BdSL) and Medical signs interpretation. Methods Considering the plethora of design criteria and balancing between rigidity and flexibility 3D models of the robotics parts are designed and 3D printed ensuring cost efficiency. With the help of modern fabrication technology, the robot is developed and assembled with proper actuators and circuitry. An image dataset is built comprising 950 images for BdSL recognition and made publicly available. We utilized the Recurrent neural network (RNN) and Convolutional neural network (CNN) for deep learning model establishment and feature extraction from video and image data. Results The developed humanoid robot has 43 Degrees of freedom (DoF) which includes two 15 DoF hands. It can imitate 16 BdSL alphabets in sign language, can capture a video or image input in real-time from the user, and recognize 10 medical signs and 38 alphabets of BdSL. The learning model for video-based medical sign recognition achieved 87.5% test accuracy. Image-based Bangla sign language recognition achieved an overall test accuracy of 98.19% in our dataset and 93.8% in another available dataset. Conclusion Compared to the state-of-the-art robotic systems for sign language interpretation, our approach has achieved higher kinematic characteristics, remarkable results in sign recognition, and impressive competency in sign imitation; all at almost 10 times lower cost than the state-of-the-art systems. The results are evidence that our approach is efficient and suitable in helping hearing and speaking impaired people. Moreover, this work initiates a research scope that can be further extended for creating equal opportunities for the hearing and speaking impaired community.

show abstract

“…In terms of RGB classification specifically, many state-of-the-art works have argued in favour of the VGG16 architecture [ 13 ] for hand gesture recognition towards sign language classification [ 14 ]. These works include British [ 15 ], American [ 16 ], Brazilian [ 17 ] and Bengali [ 18 ] Sign Languages, among others. Given the computational complexity of multimodality when visual methods are concerned in part, multimodality is a growing approach to hand gesture recognition.…”

Section: Related Workmentioning

confidence: 99%

British Sign Language Recognition via Late Fusion of Computer Vision and Leap Motion with Transfer Learning to American Sign Language

Bird

Ekárt

Faria

2020

Sensors

View full text Add to dashboard Cite

In this work, we show that a late fusion approach to multimodality in sign language recognition improves the overall ability of the model in comparison to the singular approaches of image classification (88.14%) and Leap Motion data classification (72.73%). With a large synchronous dataset of 18 BSL gestures collected from multiple subjects, two deep neural networks are benchmarked and compared to derive a best topology for each. The Vision model is implemented by a Convolutional Neural Network and optimised Artificial Neural Network, and the Leap Motion model is implemented by an evolutionary search of Artificial Neural Network topology. Next, the two best networks are fused for synchronised processing, which results in a better overall result (94.44%) as complementary features are learnt in addition to the original task. The hypothesis is further supported by application of the three models to a set of completely unseen data where a multimodality approach achieves the best results relative to the single sensor method. When transfer learning with the weights trained via British Sign Language, all three models outperform standard random weight distribution when classifying American Sign Language (ASL), and the best model overall for ASL classification was the transfer learning multimodality approach, which scored 82.55% accuracy.

show abstract

Bengali Sign Language Recognition Using Deep Convolutional Neural Network

Cited by 53 publications

References 9 publications

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network

Design and Development of a Humanoid Robot for Sign Language Interpretation

British Sign Language Recognition via Late Fusion of Computer Vision and Leap Motion with Transfer Learning to American Sign Language

Contact Info

Product

Resources

About