A Wearable Smart Glove and Its Application of Pose and Gesture Detection to Sign Language Classification

Abstract

Advances in soft sensors coupled with machine learning are enabling increasingly capable wearable systems. Since hand motion in particular can convey useful information for developing intuitive interfaces, glove-based systems can have a significant impact on many application areas. A key remaining challenge for wearables is to capture, process, and analyze data from the high-degree-of-freedom hand in real time.We propose using a commercially available conductive knit to create an unobtrusive network of resistive sensors that spans all hand joints, coupling this with an accelerometer, and deploying machine learning on a low-profile microcontroller to process and classify data. This yields a self-contained wearable device with rich sensing capabilities for hand pose and orientation, low fabrication time, and embedded activity prediction.To demonstrate its capabilities, we use it to detect static poses and dynamic gestures from American Sign Language (ASL). By pre-training a long short-term memory (LSTM) neural network and using tools to deploy it in an embedded context, the glove and an ST microcontroller can classify 12 ASL letters and 12 ASL words in real time. Using a leave-one-experiment-out cross validation methodology, networks successfully classify 96.3% of segmented examples and generate correct rolling predictions during 92.8% of real-time streaming trials.

Proximity Sensing Electronic Skin: Principles, Characteristics, and Applications

“…[ 127 ] Wearable systems find extensive use across various applications, notably in health monitoring, virtual reality, [ 128 ] and augmented reality. [ 129 , 130 ] Wearable gesture detection has been extensively researched, primarily based on resistive sensors, [ 131 , 132 ] electromyography (EMG), [ 133 , 134 , 135 ] and optical devices. [ 136 ] However, the majority of these studies focus on single‐modal systems.…”

A Review on Dynamic Hand Gesture Recognition Techniques

“…To explain neural networks, these trade-offs between scale, velocity, precision, and training could be complex and application-specific. [4] Novel graph spectral characteristics have been developed for dynamic form recognition. The proposed technique starts by preprocessing the input, such as movement of the hands, to generate a fully connected graph.…”

“…TRAINING DATA COLLECTION  DATA PROCESSING AND FEATURE EXTRACTION:  Segmentation Into Labeled Example.  Data Augmentation  Feature Extraction: Smoothing and Normalization  NEURAL NETWORK TRAINING  RESULTS AND DISCUSSION[4] …”

Assistive Data Glove for Isolated Static Postures Recognition in American Sign Language Using Neural Network

“…The good thing about sensor-based prototypes is that they are each worn and carried in public. The resultant data is normally displayed on an LCD or transmitted to the mobile or computer screen via a Bluetooth module [38,39].…”