Event-Based Gesture Recognition through a Hierarchy of Time-Surfaces for FPGA

Abstract

Neuromorphic vision sensors detect changes in luminosity taking inspiration from mammalian retina and providing a stream of events with high temporal resolution, also known as Dynamic Vision Sensors (DVS). This continuous stream of events can be used to extract spatio-temporal patterns from a scene. A time-surface represents a spatio-temporal context for a given spatial radius around an incoming event from a sensor at a specific time history. Time-surfaces can be organized in a hierarchical way to extract features from input events using the Hierarchy Of Time-Surfaces algorithm, hereinafter HOTS. HOTS can be organized in consecutive layers to extract combination of features in a similar way as some deep-learning algorithms do. This work introduces a novel FPGA architecture for accelerating HOTS network. This architecture is mainly based on block-RAM memory and the non-restoring square root algorithm, requiring basic components and enabling it for low-power low-latency embedded applications. The presented architecture has been tested on a Zynq 7100 platform at 100 MHz. The results show that the latencies are in the range of 1 μ s to 6.7 μ s, requiring a maximum dynamic power consumption of 77 mW. This system was tested with a gesture recognition dataset, obtaining an accuracy loss for 16-bit precision of only 1.2% with respect to the original software HOTS.

A High-Speed Low-Cost VLSI System Capable of On-Chip Online Learning for Dynamic Vision Sensor Data Classification

“…Our system processing latency was as short as 0.21 µs, which is 0.035 µs marginally longer than [24], and negligible to all the other implementations in Table 4. Our system realized near-perfect 99.4% and 99.3% classification accuracies on the Poker-DVS and the Posture-DVS datasets, respectively, while the works of [26,40] exhibit lower accuracies of 97.5% and 93.3% on similar DVS datasets of poker cards and gestures.…”

“…The chips of [ 24 , 26 , 27 ] were designed to accelerate the convolutional layer processing in SCNNs directly fed by DVS AER events (either from in-situ real DVS cameras or from DVS recorded datasets). The system of [ 40 ] is designed to accelerate a dedicate statistical processing method (called Hierarchy of Time-Surfaces, or HOTS) for DVS gesture classification. In Table 4 , our hardware systems realized 100 Meps DVS event throughput, about 6~128 times higher than the other DVS event processing systems in [ 24 , 26 , 27 , 40 ].…”

“…The system of [ 40 ] is designed to accelerate a dedicate statistical processing method (called Hierarchy of Time-Surfaces, or HOTS) for DVS gesture classification. In Table 4 , our hardware systems realized 100 Meps DVS event throughput, about 6~128 times higher than the other DVS event processing systems in [ 24 , 26 , 27 , 40 ]. Our system also achieved much higher energy efficiency than them, except for the one in [ 26 ] in high-efficiency low-throughput mode.…”

An Asynchronous Real-Time Corner Extraction and Tracking Algorithm for Event Camera

“…For each circle, we search for a continuous arc with higher timestamps than all other pixels on the circle. On the inner circle (blue), the arc length l inner should be within the interval of [ 3 , 6 ], and on the outer circle (yellow), the arc length l outer should be within the interval of [ 4 , 8 ] (see Figure 5 a). Alternatively, the arc length on the inner and outer circle should be within the interval of [ 10 , 13 ] and [ 12 , 16 ], respectively (see Figure 5 b).…”

“…Since there are numerous advantages, several recent works [ 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] focus on processing the unconventional output of event cameras and unlocking their potentials. In event-based vision, the corner is one of the most fundamental features, and corner event tracking is usually used in many applications, such as target tracking [ 15 , 16 ], 3D Reconstruction [ 17 , 18 ] and motion estimation [ 19 , 20 ].…”