Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

Abstract

The combination of spiking neural networks and event-based vision sensors holds the potential of highly efficient and high-bandwidth optical flow estimation. This paper presents the first hierarchical spiking architecture in which motion (direction and speed) selectivity emerges in an unsupervised fashion from the raw stimuli generated with an event-based camera. A novel adaptive neuron model and stable spike-timing-dependent plasticity formulation are at the core of this neural network governing its spike-based processing and learning, respectively. After convergence, the neural architecture exhibits the main properties of biological visual motion systems, namely feature extraction and local and global motion perception. Convolutional layers with input synapses characterized by single and multiple transmission delays are employed for feature and local motion perception, respectively; while global motion selectivity emerges in a final fully-connected layer. The proposed solution is validated using synthetic and real event sequences. Along with this paper, we provide the cuSNN library, a framework that enables GPU-accelerated simulations of large-scale spiking neural networks. Source code and samples are available at https://github.com/tudelft/cuSNN. Index Terms-Event-based vision, feature extraction, motion detection, neural nets, neuromorphic computing, unsupervised learning ! • The authors are with the Department of Control and Simulation (Micro Air Vehicle

Supervised Learning in Spiking Neural Networks

“…2 The most important factor in achieving this is to make the micro air vehicles autonomous, for which one of the main research fields is vision-based collision avoidance algorithms. For example, recent work towards this goal includes learning to predict distance using a monocular camera [16], or to estimate optical flow in an unsupervised manner using efficient neuromorphic computing [78].…”

“…In comparison, multiplicative (also called weight dependent) STDP, takes the current strength of the connection into account in its weight updates. An example of such a rule is taken from Paredes-Vallés et al [78] and simplified for illustrative purposes. The resulting weight update is shown in eq.…”

“…• cuSNN: The cuSNN library 25 is a C++ library for the design and simulation of SNNs introduced in Paredes-Vallés et al [78]. It separates itself from the other frameworks in that it is written in the potentially highly efficient CUDA 26 platform.…”

“…Some interesting and promising first results were generated using these algorithms, but none of them seem to have a thorough mathematical justification. Other works try to configure their network in such a way that the desired patterns emerge while using STDP [78,81]. Although they succeed at their respective tasks, the algorithms are not very flexible and cannot learn arbitrary input-output mappings, for that a specific error or reward signal moving back through the network is needed [6].…”

“…• Plasticity updates: This includes the algorithm for updating synaptic weights. Both from an engineering and neuroscience perspective there is decent amount of work on this topic [20,78,96,99]. An important focus point should be assuring that synaptic weights remain bounded and ideally also follow a uni-modal distribution [29,105].…”