Light-Weight Deep Learning Techniques with Advanced Processing for Real-Time Hand Gesture Recognition

Abstract

In the discipline of hand gesture and dynamic sign language recognition, deep learning approaches with high computational complexity and a wide range of parameters have been an extremely remarkable success. However, the implementation of sign language recognition applications for mobile phones with restricted storage and computing capacities is usually greatly constrained by those limited resources. In light of this situation, we suggest lightweight deep neural networks with advanced processing for real-time dynamic sign language recognition (DSLR). This paper presents a DSLR application to minimize the gap between hearing-impaired communities and regular society. The DSLR application was developed using two robust deep learning models, the GRU and the 1D CNN, combined with the MediaPipe framework. In this paper, the authors implement advanced processes to solve most of the DSLR problems, especially in real-time detection, e.g., differences in depth and location. The solution method consists of three main parts. First, the input dataset is preprocessed with our algorithm to standardize the number of frames. Then, the MediaPipe framework extracts hands and poses landmarks (features) to detect and locate them. Finally, the features of the models are passed after processing the unification of the depth and location of the body to recognize the DSL accurately. To accomplish this, the authors built a new American video-based sign dataset and named it DSL-46. DSL-46 contains 46 daily used signs that were presented with all the needed details and properties for recording the new dataset. The results of the experiments show that the presented solution method can recognize dynamic signs extremely fast and accurately, even in real-time detection. The DSLR reaches an accuracy of 98.8%, 99.84%, and 88.40% on the DSL-46, LSA64, and LIBRAS-BSL datasets, respectively.

Interpretação de gestos de Libras usando k-means e Random Forest

“…Os estudos mencionados demonstram a eficácia de diversos classificadores em reconhecer sinais Libras com altas acurácias, sobretudo em conjuntos de sinais estáticos [6]. Nos trabalhos envolvendo sinais dinâmicos [4], [9], [10], [11], [12], espera-se uma queda de acurácia, sobretudo quando se aumenta o vocabulário (número de classes). No entanto, ainda existe muito trabalho a ser feito na criação de sistemas mais robustos e eficientes que podem lidar com um grande conjunto de sinais (para aplicações mais realistas), variações na iluminação, ângulos de câmera e planos de fundo.…”

“…O uso do MediaPipe para a extração de pontos-chave do signatário permite reduzir a carga computacional necessária em comparação com a abordagem convencional de processar as imagens completas. Entretanto, diferente dos trabalhos descritos em [10], [11], nós aplicamos técnicas de agrupamento nos pontos chave e classificamos apenas os centroides dos grupos. Isso permite reduzir ainda mais o custo computacional.…”

“…Por outro lado, redes convolucionais (sobretudo as 3D) e redes recorrentes também podem exigir um poder computacional maior do que o Random Forest. Portanto, diferente dos trabalhos descritos em [4], [9], [10], [11], [12] o método de classificação empregado também permite uma redução significativa no custo computacional.…”

An Approach for Human Posture Recognition Based on the Fusion PSE-CNN-BiGRU Model

“…However, due to its binary classification, standard or not, the algorithm suits long-term posture stabilization exercises like yoga in home environments. Abdallah et al [9] extracted hand models using Mediapipe and employed a CNN network for real-time gesture detection, achieving high real-time performance but less than 89% accuracy across multiple datasets. Hong et al [10] proposed a Convolutional Neural Network-Gated Recurrent Unit (CNN-GRU) hybrid model.…”

“…In 2022, Samaan et al [13] introduced a Mediapipe-GRU method, currently limited to pose recognition but showing good real-time performance for video sequencebased recognition. Abdallah et al [9] proposed a stable but low-accuracy Mediapie-CNN-GRU pose recognition technology in 2022. In 2023, Rijayanti et al [14] developed a multi-class classification system for Mediapipe-mask R-CNN with strong generalization ability, but accuracy still requires enhancement.…”