Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network

Abstract

Sign Language Recognition is a breakthrough for communication among deaf-mute society and has been a critical research topic for years. Although some of the previous studies have successfully recognized sign language, it requires many costly instruments including sensors, devices, and high-end processing power. However, such drawbacks can be easily overcome by employing artificial intelligence-based techniques. Since, in this modern era of advanced mobile technology, using a camera to take video or images is much easier, this study demonstrates a cost-effective technique to detect American Sign Language (ASL) using an image dataset. Here, “Finger Spelling, A” dataset has been used, with 24 letters (except j and z as they contain motion). The main reason for using this dataset is that these images have a complex background with different environments and scene colors. Two layers of image processing have been used: in the first layer, images are processed as a whole for training, and in the second layer, the hand landmarks are extracted. A multi-headed convolutional neural network (CNN) model has been proposed and tested with 30% of the dataset to train these two layers. To avoid the overfitting problem, data augmentation and dynamic learning rate reduction have been used. With the proposed model, 98.981% test accuracy has been achieved. It is expected that this study may help to develop an efficient human–machine communication system for a deaf-mute society.

Sign language recognition based on dual-path background erasure convolutional neural network

“…The results in Table 4 show that our model performs well on various evaluation metrics. Relative to the methods of Refat Khan Pathan et al 36 and SHIH-HUNG YANG et al 37 , our model yields a better average test accuracy of 99.52%, which indicates that our proposed model is effective. Compared with these methods, the proposed model is lightweight.…”

“…), the proposed method in this study is very simple. The parameter sizes of the models proposed by Pathan et al 36 and Yang et al 37 are 1.88 M and 21.24 M, respectively. However, the parameter sizes of the DPCNN is 0.06 M. Therefore, the proposed model can be used in small terminals with limited resources.…”

“…Similar to gesture images, the data of these 21 key points can be used as features for training gesture recognition models 35 . Refat Khan Pathan et al 36 obtained 96.29% accuracy for the image and 98.42% accuracy for the hand features by testing the hand image and the 21 key points features of the hand separately from the "ASL Finger Spelling" 33 dataset. They then fused the two into a multi-head convolutional network and obtained a test accuracy of 98.98 percent, which was better than the two data alone.…”

Boxing behavior recognition based on artificial intelligence convolutional neural network with sports psychology assistant

“…Gangrade and Bharti 30 realized a vision-based Indian sign language gesture recognition by using CNN. Pathan et al 31 used multi-head CNN to fuse images and hand marks and sign language recognition. Zhang et al 32 used deep learning and transfer learning to realize online electromyography gesture recognition.…”

Sign language recognition based on dual-path background erasure convolutional neural network

“…Similar to gesture images, the data of these 21 key points can be used as features for training gesture recognition models [16] . Refat Khan Pathan et al [17] obtained 96.29% accuracy for the image and 98.42% accuracy for the hand features by testing the hand image and the 21 key points features of the hand separately from the "ASL Finger Spelling" [14] dataset. They then fused the two into a multi-head convolutional network and obtained a test accuracy of 98.98 percent, which was better than the two data alone.…”