Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data

Bencherif, Mohamed A.; Algabri, Mohammed; Mekhtiche, Mohamed Amine; Faisal, Mohammed; Alsulaiman, Mansour; Mathkour, Hassan; Al-Hammadi, Muneer; Ghaleb, Hamid

doi:10.1109/access.2021.3069714

Cited by 43 publications

(30 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Figure 2. The Indonesian sign movement recognition system development research series [19] Regarding the 3 rd phase, the feature extraction techniques that are examined in this work are skeleton feature extraction [20], hand shape feature extraction [21], and MobileNetV2 [22]. The skeleton and hand shape feature did not utilize deep learning models and experiments were run on another dataset, the inflectional words dataset.…”

Section: Proposed Methods For Sibi Sentence Recognition Systemmentioning

confidence: 99%

“…The skeleton and hand shape feature extraction methods yielded a best accuracy of 86.6% and 99% respectively [20], [21]. The MobileNet feature extraction [22], involving deep learning models, was running on the sentence dataset, the same dataset used by this paper. This extraction technique can pinpoint the identity for each video frame (separating epenthesis-gestures, suffixes, prefixes and root words), enabling sentence recognition from SIBI gestures with a 99% accuracy rate.…”

Section: Proposed Methods For Sibi Sentence Recognition Systemmentioning

confidence: 99%

See 1 more Smart Citation

Word recognition and automated epenthesis removal for Indonesian sign system sentence gestures

Rakun

Widhinugraha

Setyono

2022

IJEECS

View full text Add to dashboard Cite

This research <span>focuses on building a system to translate continuous Indonesian sign system (SIBI) gestures into text. In a continuous gesture, a signer will add an epenthesis (transitional) gesture, which is hand movement with no meaning but needed to connect the hand movement of one word with the next word in a continuous gesture. Reducing the number of irrelevant inputs to the model through automated epenthesis removal can improve the system's ability to recognize the words in continuous gestures. We implemented threshold conditional random fields (TCRF) to identify epenthesis gestures. The dataset consists of 2,255 videos representing 28 common sentences in SIBI. The translation system consists of MobileNetV2 as a feature extraction technique, removing epenthesis gestures found by the TCRF, and a long short-term memory (LSTM) for the classifier. With the MobileNetV2-TCRF-bidirectional LSTM model, the best word error rate (WER) and sentence accuracy (SAcc) were 33.4% and 16.2%, respectively. Intermediate-stage processing steps consisting of sandwiched majority voting of the TCRF and the removal of word labels whose number of frames is less than two frames, along with LSTM output grouping, were able to reduce WER from 33.4% to 3.4% and increase SAcc from 16.2% to 80.2%.</span>

show abstract

Section: Proposed Methods For Sibi Sentence Recognition Systemmentioning

confidence: 99%

Section: Proposed Methods For Sibi Sentence Recognition Systemmentioning

confidence: 99%

Word recognition and automated epenthesis removal for Indonesian sign system sentence gestures

Rakun

Widhinugraha

Setyono

2022

IJEECS

View full text Add to dashboard Cite

show abstract

“…Before the popularity of deep neural network, researchers focused on extracting hand-crafted features from the frames and videos, including HOG [13]- [15], hand motion trajectories [34], [35] and body joint coordinates [12]. In recent decades, convolutional neural network (CNN), including both 2D and 3D CNN, has gradually become the most common choice for feature extraction [20]- [22], [36]. Various types of CNN structures have been experimented in SLR, including 3D ResNet [37] and 3D Inception [26].…”

Section: Related Workmentioning

confidence: 99%

SignBERT: A BERT-Based Deep Learning Framework for Continuous Sign Language Recognition

2021

View full text Add to dashboard Cite

Continuous sign language recognition (CSLR) is a very challenging task in intelligent systems, since it requires to produce real-time responses while performing computationally intensive video analytics and language modeling. Previous studies mainly focus on adopting hidden Markov models or recurrent neural networks with a limited capability to model specific sign languages, and the accuracy can drop significantly when recognizing the signs performed by different signers with non-standard gestures or non-uniform speeds. In this work, we develop a deep learning framework named SignBERT, integrating the bidirectional encoder representations from transformers (BERT) with the residual neural network (ResNet), to model the underlying sign languages and extract spatial features for CSLR. We further propose a multimodal version of SignBERT, which combines the input of hand images with an intelligent feature alignment, to minimize the distance between the probability distributions of the recognition results generated by the BERT model and the hand images. Experimental results indicate that when compared to the performance of alternative approaches for CSLR, our method has better accuracy with significantly lower word error rate on three challenging continuous sign language datasets.INDEX TERMS bidirectional encoder representations from transformers, continuous sign language recognition, deep learning, video analytics I. INTRODUCTION

show abstract

“…Output size = 1+ (input size -lter size )/stride size …………….. (2) In all the situations, invariance in the translation is offered by the pooling layer, which represents the identity of a speci c object with respect to its visibility within the frame.…”

Section: Pooling Layermentioning

confidence: 99%

Machine Learning Approach for Gesture Based Arabic Sign Language Recognition for Impaired People

Ritonga

El-Aziz

Varsha³

et al. 2021

Preprint

View full text Add to dashboard Cite

Exceptional research activities have been endorsed by the Arabic Sign Language for recognizing gestures and hand signs utilizing the deep learning model. Sign languages refer to the gestures, which are utilized by hearing impaired people for communication. These gestures are complex for understanding by normal people. Due to variation of Arabic Sign Language (ArSL) from one territory to another territory or between countries, the recognition of Arabic Sign Language (ArSL) became an arduous research problem. The recognition of Arabic Sign Language has been learned and implemented utilizing multiple traditional and intelligent approaches and there were only less attempts made for enhancing the process with the help of deep learning networks. The proposed system here encapsulates a Convolutional Neural Network (CNN) based machine learning technique, which utilizes wearable sensors for recognition of the Arabic Sign Language (ArSL). The model suits to all local Arabic gestures, which are used by the hearing-impaired people of the local Arabic community. The proposed system has a reasonable and moderate accuracy. Initially a deep Convolutional network is built for feature extraction, which is extracted from the collected data by the wearable sensors. These sensors are used for recognizing accurately the 30 hand sign letters of the Arabic sign language. DG5-V hand gloves embedded with wearable sensors are used for capturing the hand movements in the dataset. The CNN approach is utilized for the classification purpose. The hand gestures of the Arabic sign language are the input and the vocalized speech is the output of the proposed system. The results achieved a recognition rate of 90%. The proposed system was found highly efficient for translating hand gestures of the Arabic Sign Language into speech and writing.

show abstract

Arabic Sign Language Recognition System Using 2D Hands and Body Skeleton Data

Cited by 43 publications

References 51 publications

Word recognition and automated epenthesis removal for Indonesian sign system sentence gestures

Word recognition and automated epenthesis removal for Indonesian sign system sentence gestures

SignBERT: A BERT-Based Deep Learning Framework for Continuous Sign Language Recognition

Machine Learning Approach for Gesture Based Arabic Sign Language Recognition for Impaired People

Contact Info

Product

Resources

About