Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints

Abstract

Pose-based approaches for sign language recognition provide light-weight and fast models that can be adopted in real-time applications. This paper presented a framework for isolated Arabic sign language (ArSL) recognition using hand and face keypoints. We employed MediaPipe pose estimator for extracting the keypoints of sign gestures in the video stream. Using the extracted keypoints, three models were proposed for sign language recognition, Long-Term Short Memory (LSTM), Temporal Convolution Networks (TCN) and Transformer based models. Moreover, we investigated the importance of non-manual features for sign language recognition systems and the obtained results showed that combining hand and face keypoints boosted the recognition accuracy by around \(4\% \) compared with only hand keypoints. The proposed models were evaluated on Arabic and Argentinian sign languages. Using the KArSL-100 dataset, the proposed pose-based Transformer achieved the highest accuracy of \(99.74\% \) and \(68.2\% \) in signer-dependent and independent modes, respectively. Additionally, the Transformer was evaluated on the LSA64 dataset and obtained an accuracy of \(98.25\% \) and \(91.09\% \) in signer-dependent and independent modes, respectively. Consequently, the pose-based Transformer outperformed the state-of-the-art techniques on both datasets using keypoints from the signer’s hands and face.

Signer-Independent Arabic Sign Language Recognition System Using Deep Learning Model

“…Another study [ 35 ] used a larger dataset of 100 classes and pose-based transformers for Arabic Sign Language recognition. The authors also used signer-independent mode for evaluation, as we did.…”

“…However, they only achieved an accuracy of 68.2%, while we achieved 87.69%. Additionally, the KArSL-100 dataset used in [ 35 ] was recorded from only three subjects, while our dataset had data from four subjects. This suggests that our proposed model is more robust to variations in signers’ hand movements.…”

“…The model achieved an average word error rate (WER) of 0.50. In Reference [ 35 ], a study by the author introduced a pose-based Transformer model for the purpose of recognizing a more extensive dataset called KArSL-100. This dataset encompassed 100 distinct classes and was specifically designed for the task of sign video recognition.…”

“…This dataset encompassed 100 distinct classes and was specifically designed for the task of sign video recognition. The researchers in Reference [ 35 ] attained an accuracy rate of 68.2% in signer-independent mode. Similarly, the mArSL dataset of 6748 videos of 50 classes performed by 4 signers was proposed in [ 16 ] which is a multi-modality dataset containing RGB videos, depth data, skeleton joint points, and face information of manual and non-manual signs.…”

Efficient YOLO Based Deep Learning Model for Arabic Sign Language Recognition

“…They also developed an encoder-decoder model for recognizing sign language sentences and achieved a word error rate (WER) of 0.50 on average. The authors [26] introduced a Transformer model based on stance, specifically tailored for the KArSL-100 dataset. This dataset consists of 100 different classes and is focused on recognizing sign videos and were able to attain a 68.2% accuracy rate while using a signer-independent mode.…”