American Sign Language Alphabet Recognition Using Inertial Motion Capture System with Deep Learning

Abstract

Sign language is designed as a natural communication method for the deaf community to convey messages and connect with society. In American sign language, twenty-six special sign gestures from the alphabet are used for the fingerspelling of proper words. The purpose of this research is to classify the hand gestures in the alphabet and recognize a sequence of gestures in the fingerspelling using an inertial hand motion capture system. In this work, time and time-frequency domain features and angle-based features are extracted from the raw data for classification with convolutional neural network-based classifiers. In fingerspelling recognition, we explore two kinds of models: connectionist temporal classification and encoder-decoder structured sequence recognition model. The study reveals that the classification model achieves an average accuracy of 74.8% for dynamic ASL gestures considering user independence. Moreover, the proposed two sequence recognition models achieve 55.1%, 93.4% accuracy in word-level evaluation, and 86.5%, 97.9% in the letter-level evaluation of fingerspelling. The proposed method has the potential to recognize more hand gestures of sign language with highly reliable inertial data from the device.

Spatial–temporal attention with graph and general neural network-based sign language recognition

“…Here, for the spatial mask cases, we assign 1 for the spatial position and set 0 for the temporal position, aiming to block the temporal value and only pass the spatial position value. We updated the value of the weight matrix by manual setting aiming to perform spatial mask M S and temporal mask M T , using the following formula Equation ( 11) and (12).…”

“…The 3D hand joint data is typically obtained using affordable depth cameras like Microsoft Kinect, Intel RealSense, or Microsoft Oak-D, which enhance hand pose estimation accuracy [3,10]. Some researchers have employed traditional methods to extract powerful features from hand joints [12][13][14], achieving high performance in some instances but still facing limitations in generalization capabilities. Recently, researchers have turned to deep learning algorithms to improve performance accuracy [15][16][17].…”