Toward Robust Arabic Sign Language Recognition via Vision Transformers and Local Interpretable Model-agnostic Explanations Integration

Abstract

People with severe or substantial hearing loss find it difficult to communicate with others. Poor communication can have a significant impact on the mental health of deaf people. For individuals who are deaf or hard of hearing, sign language (SL) is the major mode of communication in their daily life. Motivated by the need to develop robust and interpretable models for the deaf community, this study presents a computer-aided diagnosis (CAD) framework for Arabic SL recognition. The interpretability and management of complicated spatial connections in SL images have been limited by prior studies using convolutional neural networks. To improve accuracy and offer model transparency, the proposed CAD framework incorporates state-of-the-art technologies such as local interpretable model-agnostic explanations (LIME) and vision transformers (ViTs). ViTs use self-attention mechanisms to interpret visuals in SL, capturing global dependencies. A stacking/voting strategy is then used to aggregate predictions from many ViT models, further optimizing the system. Two large datasets, the “ArSL21L: Arabic Sign Language Letter Dataset” and the “RGB Arabic Alphabets Sign Language Dataset,” totaling over 22,000 pictures, were used to validate this approach. Metrics including intersection over union, balanced accuracy, Youden’s index, Yule’s Q, F1 score, accuracy, precision, recall, and specificity were used to assess performance. The results show that the stacking method, which makes use of many ViT models, outperforms traditional models in every performance indicator and achieves an impressive accuracy of 99.46% and 99.88% on the ArSL21L and RGB datasets, respectively. For practical applications, interpretability is ensured by using LIME, which offers clear visual explanations for the model’s predictions.