A Novel Illumination-Robust Hand Gesture Recognition System With Event-Based Neuromorphic Vision Sensor

Chen, Guang; Xu, Zhongcong; Li, Zhijun; Tang, Huajin; Qu, Sanqing; Ren, Kejia; Knoll, Alois

doi:10.1109/tase.2020.3045880

Cited by 29 publications

(12 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Schraml et al [30] used a stereoscopic system to track people, while Piatkowska et al [46] addressed the problem of tracking people in high-occlusion environments through the use of Gaussian Mixture Models (GMM) [47]; this clustering method is succesfully used in [48] to track pedestrians. Meanwhile, [49] addresses the pedestrian detection problem with an event-to-frame encoding method combined with Convolutional Neural Networks (CNN). Recently, the NeuroAED system has been presented; it aims to efficiently detect abnormal events in visual surveillance [50].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Adaptive Multi-Object Tracking-by-Clustering Algorithm With a Bio-Inspired System

et al. 2022

View full text Add to dashboard Cite

A problem of current interest is how to emulate nature by acquiring information in a neuromorphic-like fashion; namely, by using configurable hardware and electronic systems to emulate the information gathering and processing strategies of biological systems. In this paper, we introduce Bio-CAMSHIFT, an algorithm for a bio-inspired system that acquires information via a neuromorphic process and uses it to track multiple objects. The system consists of a silicon retina that simulates the behavior of the human eye together with a communication system that uses an Address-Event Representation protocol to transmit information in a way analogous to that of biological neural systems. An unsupervised procedure, based on the CAMSHIFT algorithm, is then used for multi-object tracking. It takes advantage of the retina's high event rate to adapt to the changing sizes of the objects in its field of view. The proposed system has been experimentally validated using a data set from Freeway 210 in Pasadena, California, demonstrating a significantly better improvement in terms of multi-vehicle detection and tracking performance over the current state of the art.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Recently, in [59] a spatial-temporal mixed particle filter (SMP Filter) is proposed to track LED-based rectangles. In [49], a Restricted Spatiotemporal Particle Filter (RSPF) tracking algorithm is presented, and evaluated tracking fingers. Lastly, in [60], a combined use of SNNs and silicon retina is proposed, applied to object tracking.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Adaptive Multi-Object Tracking-by-Clustering Algorithm With a Bio-Inspired System

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The existing work on neuromorphic vision sensing in computer vision can be grouped into three themes: object detection [47]- [49], pedestrian detection [50], [51] and hand gesture recognition [33]- [35]. There is only a little work on exploring the neuromorphic data beyond object detection addressing highly semantic applications, such as, multi class action recognition, which still poses an important challenge.…”

Section: Related Workmentioning

confidence: 99%

Making Sense of Neuromorphic Event Data for Human Action Recognition

2021

View full text Add to dashboard Cite

Neuromorphic vision sensors provide low power sensing and capture salient spatial-temporal events. The majority of the existing neuromorphic sensing work focus on object detection. However, since they only record the events, they provide an efficient signal domain for privacy aware surveillance tasks. This paper explores how the neuromorphic vision sensor data streams can be analysed for human action recognition, which is a challenging application. The proposed method is based on handcrafted features. It consists of a pre-processing step for removing the noisy events followed by the extraction of handcrafted local and global feature vectors corresponding to the underlying human action. The local features are extracted considering a set of high-order descriptive statistics from the spatio-temporal events in a time window slice, while the global features are extracted by considering the frequencies of occurrences of the temporal event sequences. Then, low complexity classifiers, such as, support vector machines (SVM) and K-Nearest Neighbours (KNN), are trained using these feature vectors. The proposed method evaluation uses three groups of datasets: Emulator-based, re-recording-based and native NVS-based. The proposed method has outperformed the existing methods in terms of human action recognition accuracy rates by 0.54%, 19.3% and 25.61% for E-KTH, E-UCF11 and E-HMDB51 datasets, respectively. This paper also reports results for three further datasets: E-UCF50, R-UCF50, and N-Actions, which are reported for the first time for human action recognition on neuromorphic vision sensor domain.INDEX TERMS Neuromorphic vision sensing (NVS), event cameras, dynamic vision sensing (DVS), Human action recognition (HAR), Local features, Global features,

show abstract

“…However, the pixels in the hemispherical image sensors consist of photodetectors without the functions of information memory and preprocessing; the ORAM requires the operation processes of optically SET and electrically RESET; NN vision sensors that use 2D materials are not readily compatible with large-scale integration. Moreover, although much progress has been made in the efficient neuromorphic processing of electrical or optical signals, − methods to convert optical images to electrical signals in real time must still be improved, particularly for time-critical applications. Electrohydrodynamic printing is an attractive technique to fabricate well-aligned and well-patterned semiconductor fibers.…”

Section: Introductionmentioning

confidence: 99%

Bio-Inspired Vision and Neuromorphic Image Processing Using Printable Metal Oxide Photonic Synapses

Sun

et al. 2022

ACS Photonics

View full text Add to dashboard Cite

We propose a neuromorphic vision system that uses digitally printed metal oxide photonic synapses with optical tunability and temporally correlated plasticity. The neuromorphic vision system provides the first demonstration of encoding ambient light intensity in the time domain and captures optical images by encoding their pixel intensity into pulsatile signals in real time, analogous to a biological visual retina and optic nerve. The system can then process the information and form memory, emulating the image data processing in the human brain. The system can realize image-preprocessing functions to increase image contrast and reduce image background noise, thereby effectively improving the classification and recognition accuracy. Dynamic stimulation of visual cortical prosthetics shows that the system is capable of dynamic perception and memory to detect motion and mimic coherent visual perception. Furthermore, the photonic synapses with a two-terminal structure using printed semiconducting fiber arrays could also facilitate large-scale integration. The proposed system offers the potential to simplify neuromorphic visual circuits and may have applications in autonomous smart devices, such as driverless cars, smart surveillance systems, and intelligent healthcare.

show abstract

A Novel Illumination-Robust Hand Gesture Recognition System With Event-Based Neuromorphic Vision Sensor

Cited by 29 publications

References 49 publications

Unsupervised Adaptive Multi-Object Tracking-by-Clustering Algorithm With a Bio-Inspired System

Unsupervised Adaptive Multi-Object Tracking-by-Clustering Algorithm With a Bio-Inspired System

Making Sense of Neuromorphic Event Data for Human Action Recognition

Bio-Inspired Vision and Neuromorphic Image Processing Using Printable Metal Oxide Photonic Synapses

Contact Info

Product

Resources

About