Real-Time Gesture Interface Based on Event-Driven Processing From Stereo Silicon Retinas

Abstract

We propose a real-time hand gesture interface based on combining a stereo pair of biologically inspired event-based dynamic vision sensor (DVS) silicon retinas with neuromorphic event-driven postprocessing. Compared with conventional vision or 3-D sensors, the use of DVSs, which output asynchronous and sparse events in response to motion, eliminates the need to extract movements from sequences of video frames, and allows significantly faster and more energy-efficient processing. In addition, the rate of input events depends on the observed movements, and thus provides an additional cue for solving the gesture spotting problem, i.e., finding the onsets and offsets of gestures. We propose a postprocessing framework based on spiking neural networks that can process the events received from the DVSs in real time, and provides an architecture for future implementation in neuromorphic hardware devices. The motion trajectories of moving hands are detected by spatiotemporally correlating the stereoscopically verged asynchronous events from the DVSs by using leaky integrate-and-fire (LIF) neurons. Adaptive thresholds of the LIF neurons achieve the segmentation of trajectories, which are then translated into discrete and finite feature vectors. The feature vectors are classified with hidden Markov models, using a separate Gaussian mixture model for spotting irrelevant transition gestures. The disparity information from stereovision is used to adapt LIF neuron parameters to achieve recognition invariant of the distance of the user to the sensor, and also helps to filter out movements in the background of the user. Exploiting the high dynamic range of DVSs, furthermore, allows gesture recognition over a 60-dB range of scene illuminance. The system achieves recognition rates well over 90% under a variety of variable conditions with static and dynamic backgrounds with naïve users.

A Motion-Based Feature for Event-Based Pattern Recognition

“…Gesture recognition using neuromorphic camera has already been investigated by Lee et al (2014). A stereo pair of DVS allows them to compute disparity in order to cluster the hand.…”

“…This strong interest in such a sensor is essentially due to its ability to provide visual information as a high temporal resolution, luminance-free, and non-redundant stream. This makes it a fitting for high-speed applications [e.g., gesture recognition as in Lee et al (2014), high-speed object tracking as in Lagorce et al (2014), Mueggler et al (2015a)].…”

Applications of event‐based image sensors—Review and analysis

“…It can be integrated into the sensors of a smart house. Gesture recognition is also possible Robotics integration, control, and navigation .…”

Inceptive Event Time-Surfaces for Object Classification Using Neuromorphic Cameras

“…arXiv:2002.11656v1 [cs.NE] 26 Feb 2020 these neuromorphically inspired cameras can operate at extremely high temporal resolution (>800kHz), low latency (20 microseconds), wide dynamic range (> 120dB), and low power (30mW). They report only changes in the pixel intensity, requiring a new set of techniques to perform basic image processing and computer vision tasks-examples include optical flow [3,8], feature extraction [4,12,13], gesture recognition [2,11], and object recognition [5,14].…”