Korean Sign Language Recognition Using Transformer-Based Deep Neural Network

Abstract

Sign language recognition (SLR) is one of the crucial applications of the hand gesture recognition and computer vision research domain. There are many researchers who have been working to develop a hand gesture-based SLR application for English, Turkey, Arabic, and other sign languages. However, few studies have been conducted on Korean sign language classification because few KSL datasets are publicly available. In addition, the existing Korean sign language recognition work still faces challenges in being conducted efficiently because light illumination and background complexity are the major problems in this field. In the last decade, researchers successfully applied a vision-based transformer for recognizing sign language by extracting long-range dependency within the image. Moreover, there is a significant gap between the CNN and transformer in terms of the performance and efficiency of the model. In addition, we have not found a combination of CNN and transformer-based Korean sign language recognition models yet. To overcome the challenges, we proposed a convolution and transformer-based multi-branch network aiming to take advantage of the long-range dependencies computation of the transformer and local feature calculation of the CNN for sign language recognition. We extracted initial features with the grained model and then parallelly extracted features from the transformer and CNN. After concatenating the local and long-range dependencies features, a new classification module was applied for the classification. We evaluated the proposed model with a KSL benchmark dataset and our lab dataset, where our model achieved 89.00% accuracy for 77 label KSL dataset and 98.30% accuracy for the lab dataset. The higher performance proves that the proposed model can achieve a generalized property with considerably less computational cost.

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

“…In the traditional VIT architecture, input signals are divided into non-overlapping patches and mapped to the input dimension through linear projection. However, modeling the structural information within local patches solely through linear projection can be less effective [38], as it overlooks local relationships and internal structural information within the patches [39]. In the field of video representation learning, one approach is to aggregate pixel-level features for each frame by adding one-dimensional convolution (1D-CNN) to capture temporal clues between the same spatial positions [40].…”

Dynamic Doubled-handed sign language Recognition for deaf and dumb people using Vision Transformers

“…The Korean SL (KSL) dataset is useful for real-life scenarios. SL data are selected from 20 persons around 17 locations [34]. Also, the dataset gathered the signers' facial expressions along with their hand gesticulation.…”

“…In this subsection, the proposed method's effectiveness is compared against various state-of-theart approaches, such as Meta-ELM [12], SVM-CNN [17], LSTMRNN-KNN [18], T-CNN [19], TDDRMN [25], and PSO-CNN [27]. The results of the comparative study are detailed below.…”

Combining State-of-the-Art Pre-Trained Deep Learning Models: A Noble Approach for Skin Cancer Detection Using Max Voting Ensemble

“…Consequently, there has been a rising interest in leveraging machine learning methods, notably deep learning techniques, to enhance the precision and efficiency of skin cancer diagnosis. Over the past few years, deep learning methods [ 26 ], particularly convolutional neural networks (CNNs) [ 17 , 23 , 26 , 27 , 28 ], have demonstrated significant potential for accurately identifying and categorizing skin cancer from medical images of skin lesions [ 7 , 16 , 24 , 25 ]. In this review of relevant literature, we explore recent studies focusing on the detection and classification of skin cancer through the application of deep learning and transfer learning methodologies.…”