Identification of Static and Dynamic Signs of the Mexican Sign Language Alphabet for Smartphones using Deep Learning and Image Processing

Abstract

The Mexican Sign Language (MSL) is a language with its own syntax and lexicon. It is used by the deaf people, who use it to express thoughts, ideas and emotions. However, most of hearing people are unable to understand this language. The alphabet of any Sign Language (SL) is composed of signs where each sign corresponds to a letter of the alphabet of the dominant language in the region, for example, Spanish or English. Most signs of a signed alphabet are static, that means, they are only composed by the configuration of the hands. However, there are letters that are represented by signs that include movement. The present work proposes a system that, using artificial vision techniques and image processing, identify the 27 letters -including dynamic and static signs-of the Spanish alphabet in a mobile application. To solve the problem of sign identification it was used a combination of image processing techniques and deep learning. Canny and Camshift algortihms was implemented for the recognition of edges and trajectories in signs with movement. Once the characteristics were identified, the K-means and Tensorflow algorithms were used to classify the signs. The system achieves a 92% accuracy in the alphabet sign detection.

Spatial–temporal attention with graph and general neural network-based sign language recognition

“…However, this sensor-based work has been limited to hand processing. Last decades, researchers were again focused on RGB cameras to collect the hand skeleton information of the hand gesture and hand signs [4,32,[47][48][49][50][51]. Many researchers have developed camera data-based hand gesture recognition using conventional machine learning algorithms.…”

Towards a Bidirectional Mexican Sign Language–Spanish Translation System: A Deep Learning Approach

“…Parallel to these developments, Martínez-Gutiérrez et al [16] and Martinez-Seis et al [17] focused on MSL alphabet recognition through advanced computational methods, each achieving notable accuracy in their respective areas.…”

Hacia la construcción de una base de datos abierta de la LSM

“…reconocimiento de señas mediante aprendizaje profundo (Rao et al, 2018;Xia et al, 2022;Kothadiya et al, 2022); sin embargo, hay poca investigación específica de la LSM y la mayoría se ha centrado en el recono-cimiento estático de la LSM (Solís-V et al, 2014;Solís et al, 2015Carmona-Arroyo et al, 2021;Rios-Figueroa et al, 2022). En el reconocimiento de señas estáticas y dinámicas de la LSM se encuentra (Martinez-Seis et al, 2019). Adicionalmente, existe reconocimiento de vocabulario de la LSM usando Leap Motion Controller (Nájera et al, 2016) y desarrollo de traductores de habla a señas (español a LSM) usando sensores RGBD (Trujillo-Romero y Caballero-Morales, 2012; Caballero-Morales y Trujillo-Romero, 2013) y de señas a habla (LSM a español) (Garcia-Bautista et al, 2016).…”

“…Este enfoque permitiría identificar y delimitar las regiones de interés. Otra perspectiva prometedora es el seguimiento de la trayectoria de las manos mediante algoritmos, tales como el algoritmo de Camshift, (Bradski, 1998), cuya eficacia ha sido probada en investigaciones previas (Martinez-Seis et al, 2019). La aplicación de dicha técnica podría proporcionar información valiosa acerca de la evolución y el desplazamiento de los objetos de interés a lo largo de la escena.…”