Human-Computer Interaction with Hand Gesture Recognition Using ResNet and MobileNet

Abstract

Sign language is the native language of deaf people, which they use in their daily life, and it facilitates the communication process between deaf people. The problem faced by deaf people is targeted using sign language technique. Sign language refers to the use of the arms and hands to communicate, particularly among those who are deaf. This varies depending on the person and the location from which they come. As a result, there is no standardization about the sign language to be used; for example, American, British, Chinese, and Arab sign languages are all distinct. Here, in this study we trained a model, which will be able to classify the Arabic sign language, which consists of 32 Arabic alphabet sign classes. In images, sign language is detected through the pose of the hand. In this study, we proposed a framework, which consists of two CNN models, and each of them is individually trained on the training set. The final predictions of the two models were ensembled to achieve higher results. The dataset used in this study is released in 2019 and is called as ArSL2018. It is launched at the Prince Mohammad Bin Fahd University, Al Khobar, Saudi Arabia. The main contribution in this study is resizing the images to 64 ∗ 64 pixels, converting from grayscale images to three-channel images, and then applying the median filter to the images, which acts as lowpass filtering in order to smooth the images and reduce noise and to make the model more robust to avoid overfitting. Then, the preprocessed image is fed into two different models, which are ResNet50 and MobileNetV2. ResNet50 and MobileNetV2 architectures were implemented together. The results we achieved on the test set for the whole data are with an accuracy of about 97% after applying many preprocessing techniques and different hyperparameters for each model, and also different data augmentation techniques.

Innovative integration of multi-scale residual networks and MK-MMD for enhanced feature representation in fault diagnosis

“…These cross-layer connections enable gradients to propagate smoothly in the network and can be back-propagated directly from the end to the beginning. This method effectively solves the problem of vanishing gradients when the network becomes deeper, improving both the training speed and performance of the model [30][31][32][33].…”

Red Deer Optimization with Deep Learning Enabled Agricultural Plant Disease Detection and Classification Model

“…In this research, the MobileNet-v3 technique is exploited to derive feature vectors. According to the infrastructure which utilizes depthwise separable convolutional for creating lightweight DNNs, calculation, and size trade-offs of lessening the MobileNetV2 structure by width multiplier [20]. Typical convolutional is at a high price as every function was multiplication.…”

A Lightweight Multi-Person Pose Estimation Scheme Based on Jetson Nano

“…Chen et al (2018) use a feature pyramid network to localize simple joints and a refining network that integrates features of all levels from the previous network to handle complex joints. A simple-structured network (Xiao et al, 2018) with ResNet (Alnuaim et al, 2022) as the backbone and a few deconvolutional layers as upsampling head show effective and competitive results. Sun et al (2019) present a robust high-resolution network, where a high-resolution subnetwork is established in the first stage, and high-to-low-resolution subnetworks are added one by one in parallel in subsequent steps, conducting repeated multiscale feature fusions.…”