Soft Optoelectronic Sensors with Deep Learning for Gesture Recognition

Abstract

With the rapid development of deep learning and computing power, human–computer interactions, and interfaces are attracting attentions in industrial and academic research. Flexible human–computer interaction can greatly improve productivity and enable robots to work in extreme environments that humans cannot tolerate. The research of gesture recognition is emerging and provides a new way of studying the human–computer interactions. However, compared with the entire human body, human hands are dexterous organs with more complex and flexible joints, which makes hand gesture recognition a challenging problem. Here, a robust and cost‐effective gesture recognition system is reported through the soft optoelectronic sensors. An array of polymer‐encapsulated U‐shaped microfiber (UMF) attached to a glove is fabricated for sensitive finger motion detection. The anisotropic strain response of UMF is measured with a sensitivity of 15.98 (2.20) in the x‐direction (y‐direction). A deep learning network (VGGNet) is developed to process the optical signals for analyzing and classifying hand gestures. The experiments show that VGGNet has high recognition accuracy of 99.2% for the test datasets with ten classified gestures. This work provides a potential optical interface in studying gesture recognition and biomechanical signatures, which can also be applied in virtual reality systems and interactive game platforms.

Optical‐Nanofiber‐Enabled Gesture‐Recognition Wristband for Human–Machine Interaction with the Assistance of Machine Learning

“…[26] Already, optical-micro/ nanofiber-based sensors have demonstrated high sensitivity, fast response, and a tunable working range for pressure sensing, [27,28] strain sensing, [29,30] and bending-angle monitoring. [31][32][33] A highly sensitive sensor for obtaining mechanical information from the surface muscles of the hand would improve the accuracy of a GRW and reduce the number of required sensing units. A single gesture typically generates different signals at different sensors; the gesture can be recognized by combining basic signal-processing methods with a machine-learning algorithm.…”

“…[ 26 ] Already, optical‐micro/nanofiber‐based sensors have demonstrated high sensitivity, fast response, and a tunable working range for pressure sensing, [ 27,28 ] strain sensing, [ 29,30 ] and bending‐angle monitoring. [ 31–33 ]…”

Soft Polymer Optical Fiber Sensors for Intelligent Recognition of Elastomer Deformations and Wearable Applications

“…However, most of these electrical sensors struggle to operate in harsh industrial conditions that involve humidity and chemical corrosion, and they also face challenges in prolonged operations in complex electromagnetic environments [3,7,10,11]. Soft and wearable sensing devices based on optical waveguides (optical fibers) have emerged recently, and compared with their electronic counterpart, optical sensors have anti-electromagnetic interference and electrical capabilities [10][11][12][13][14][15][16][17][18][19][20]. In addition, optical sensing systems have the characteristics of a large transmission bandwidth, multiplexing, and diversified modulation/demodulation [11,21].…”

Ultra-fast 3D printing of assembly—free complex optics with sub-nanometer surface quality at mesoscale

“…Optical devices with excellent illumination, sensing, light modulating, and display capabilities [1] can be directly applied to sensors [2], hologram reconstruction [3], human-computer interaction [4], and other optoelectronics [5]. Traditional manufacturing methods for optical devices primarily comprise precision machining [6], casting, and injection molding [7].…”