Embodied Neuromorphic Vision with Continuous Random Backpropagation

Abstract

Spike-based communication between biological neurons is sparse and unreliable. This enables the brain to process visual information from the eyes efficiently. Taking inspiration from biology, artificial spiking neural networks coupled with silicon retinas attempt to model these computations. Recent findings in machine learning allowed the derivation of a family of powerful synaptic plasticity rules approximating backpropagation for spiking networks. Are these rules capable of processing real-world visual sensory data? In this paper, we evaluate the performance of Event-Driven Random Back-Propagation (eRBP) at learning representations from event streams provided by a Dynamic Vision Sensor (DVS). First, we show that eRBP matches state-of-the-art performance on the DvsGesture dataset with the addition of a simple covert attention mechanism. By remapping visual receptive fields relatively to the center of the motion, this attention mechanism provides translation invariance at low computational cost compared to convolutions. Second, we successfully integrate eRBP in a real robotic setup, where a robotic arm grasps objects according to detected visual affordances. In this setup, visual information is actively sensed by a DVS mounted on a robotic head performing microsaccadic eye movements. We show that our method classifies affordances within 100ms after microsaccade onset, which is comparable to human performance reported in behavioral study. Our results suggest that advances in neuromorphic technology and plasticity rules enable the development of autonomous robots operating at high speed and low energy consumption.

A visual-olfactory multisensory fusion spike neural network for early fire/smoke detection

“…2c. SNN model using eRBP algorithm has been proved to have good performance in the application of dynamic image recognition 20,23 . The spiketiming-dependent plasticity (STDP) learning mechanism is used for the generation of gas feature vectors, which has been introduced for gas recognition and proved applicable 21 .…”

A perspective on large-scale simulation as an enabler for novel biorobotics applications

“…The Neurorobotics Platform (NRP) (Falotico et al, 2017), developed within the context of the Human Brain Project, has formed as the basis of many fruitful experiments in neuroscience of rehabilitation (Allegra Mascaro et al, 2020), biologically-inspired robot control (Capolei et al, 2019;Angelidis et al, 2021), dynamic vision systems based on event-based cameras (Bornet et al, 2019;Kaiser et al, 2020), robot manipulation (Bing et al, 2021), pattern recognition (Galindo et al, 2020) among others. It is one of the few simulation engines that enable the interaction of SNNs with virtual robots, closing the loop between neuroscience and robotics simulation.…”