Signer-Independent Arabic Sign Language Recognition System Using Deep Learning Model

Abstract

Every one of us has a unique manner of communicating to explore the world, and such communication helps to interpret life. Sign language is the popular language of communication for hearing and speech-disabled people. When a sign language user interacts with a non-sign language user, it becomes difficult for a signer to express themselves to another person. A sign language recognition system can help a signer to interpret the sign of a non-sign language user. This study presents a sign language recognition system that is capable of recognizing Arabic Sign Language from recorded RGB videos. To achieve this, two datasets were considered, such as (1) the raw dataset and (2) the face–hand region-based segmented dataset produced from the raw dataset. Moreover, operational layer-based multi-layer perceptron “SelfMLP” is proposed in this study to build CNN-LSTM-SelfMLP models for Arabic Sign Language recognition. MobileNetV2 and ResNet18-based CNN backbones and three SelfMLPs were used to construct six different models of CNN-LSTM-SelfMLP architecture for performance comparison of Arabic Sign Language recognition. This study examined the signer-independent mode to deal with real-time application circumstances. As a result, MobileNetV2-LSTM-SelfMLP on the segmented dataset achieved the best accuracy of 87.69% with 88.57% precision, 87.69% recall, 87.72% F1 score, and 99.75% specificity. Overall, face–hand region-based segmentation and SelfMLP-infused MobileNetV2-LSTM-SelfMLP surpassed the previous findings on Arabic Sign Language recognition by 10.970% accuracy.

Efficient YOLO-Based Deep Learning Model for Arabic Sign Language Recognition

“…ANFIS has been proven pioneering by many researchers; however, it is counteracted by its significant weakness in swiftly adapting and performing intricate tasks of gesture recognition, an essential criterion in any sign language recognition system. As a result, ANFIS has achieved a high precision of 86.69%, but its performance and adaptation ability to intricate gesture recognition tasks remain limited (Podder et al, 2023). A detailed comparison of the proposed model with SOTA approaches is presented with an accuracy of 94.46% (Aldhahri et al, 2023) presented in (Table 1).…”

Enhanced Weak Spatial Modeling Through CNN-Based Deep Sign Language Skeletal Feature Transformation

“…The transformer is trained from augmented MediaPipe poses + 33 landmarks and returns an accuracy of 68.2% from user independence mode. Podder et al [53] proposed features from the face-hand region-based segmentation and SelfMLPinfused MobileNetV2-LSTM-SelfMLP. Overall accuracy of 88.57%.…”

“…We loop through each C in the SFC. We [53] Light model Limited data set MobileNetV2-LSTM-SelfMLP (q = 3) 88.57 Alyami et al [52] Sequential learning Low accuracy Transformer-based poses + landmarks SD 99.7 and SI 68.2 Balaha et al [34] Dynamic ArSLR Limited data set 20 ArSL words and hybrid CNN-RNN 98 AlSulaiman et al [54] Effective image modeling Limited data set 3D-GCN vertices and edges 97.25 Hany et al [35] Novel ArSL characters Letters only Augmented Q-CNN-based features 99.54 at 42 min Aldhahri et al [36] Light…”

Sign Language Word Detection Using LRCN

“…In the field of image processing, there has been an increase in popularity and pervasive adoption of deep learning and machine learning techniques [7,8]. Deep learning and computer vision can also be employed to help advance this goal, ensuring ease of use [9,10]. A Recurrent Neural Network is a form of neural network that incorporates loops for internal data storage [11].…”