A Prototype for Mexican Sign Language Recognition and Synthesis in Support of a Primary Care Physician

Abstract

Few hearing people know and use Mexican Sign Language (MSL). Consequently, this is the main barrier between deaf and hearing people. This study proposes a system that recognizes and animates in real time a set of signs belonging to the semantic field of a general medicine consultation service. Therefore, a linkage between a hearing doctor and a deaf patient can be established in a non-intrusive way and with an easy dynamic interaction. Our main contribution is a bidirectional translator system for Mexican sign language in the context of a general medical consultation. The recognition module uses a MS Kinect sensor to obtain the sign trajectories, and images to feed hidden Markov Models (HMMs) for processing signs samples in real time. The experiments show recognition of 82 different signs from 22 participants. As a result, accuracy and F1 scores average rates of 99% and 88% were obtained.

Towards a Bidirectional Mexican Sign Language–Spanish Translation System: A Deep Learning Approach

“…In a similar vein, focusing on a specific application area, Sosa-Jimenez et al [14] developed a research prototype tailored for primary care health services in Mexican Sign Language. Their use of Microsoft Kinect sensors [15] and HMMs highlights the trend of specialized systems addressing distinct contexts like healthcare.…”

RF-CSign: A Chinese Sign Language Recognition System Based on Large Kernel Convolution and Normalization-Based Attention

“…Studies have used RGB cameras to capture sign language videos and then recognize sign language through bidirectional VTNs [29]. In addition, certain prior studies used Kinect [10], [30] and Leap Motion [31] to collect data on sign language, which produced better recognition results. For instance, Sun et al [30] used a Kinect device to collect a large number of consecutive Chinese sign languages and achieved accurate sign language image matching using an Extenics Immune Neural Net.…”

“…𝑁 𝑇𝑃 + 𝑁 𝑇𝑁 𝑁 𝑇𝑃 + 𝑁 𝑇𝑁 + 𝑁 𝐹𝑃 + 𝑁 𝐹𝑁 (10) Concurrently, the F1-score is presented as the balanced average of precision (P) and recall (R) within the model:…”

“…Approaches to sign language recognition can be classified as vision-based [9], [10], sensor-based [11], [12], [13], [14], [15], [16], [17], and a blend of both [18]. Vision-based gesture recognition techniques use a camera to capture the visual data of a gesture, and then process the visual data to complete the recognition.…”