A Spiking Neural Network Model of Depth from Defocus for Event-based Neuromorphic Vision

Abstract

Depth from defocus is an important mechanism that enables vision systems to perceive depth. While machine vision has developed several algorithms to estimate depth from the amount of defocus present at the focal plane, existing techniques are slow, energy demanding and mainly relying on numerous acquisitions and massive amounts of filtering operations on the pixels’ absolute luminance value. Recent advances in neuromorphic engineering allow an alternative to this problem, with the use of event-based silicon retinas and neural processing devices inspired by the organizing principles of the brain. In this paper, we present a low power, compact and computationally inexpensive setup to estimate depth in a 3D scene in real time at high rates that can be directly implemented with massively parallel, compact, low-latency and low-power neuromorphic engineering devices. Exploiting the high temporal resolution of the event-based silicon retina, we are able to extract depth at 100 Hz for a power budget lower than a 200 mW (10 mW for the camera, 90 mW for the liquid lens and ~100 mW for the computation). We validate the model with experimental results, highlighting features that are consistent with both computational neuroscience and recent findings in the retina physiology. We demonstrate its efficiency with a prototype of a neuromorphic hardware system and provide testable predictions on the role of spike-based representations and temporal dynamics in biological depth from defocus experiments reported in the literature.

Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms

“…As a result, it is very progressive and combines many benefits of the research it is based on. Also noteworthy are the results of Martel et al (2018) and Haessig et al (2019). Hereby, active approaches requiring additional hardware are introduced.…”

“…Quite different solutions to recover depth from event-based data are shown in Martel et al (2018) and Haessig et al (2019). These are active techniques and require additional hardware, setting them apart from most investigated methods.…”

“…These are active techniques and require additional hardware, setting them apart from most investigated methods. In Haessig et al (2019), the known method to estimate depth from the amount of defocus at the focal plane is transferred to neuromorphic sensors and hardware. This is a simple yet elegant solution, whereby the camera alters its focal distance in a steady manner and a SNN computes the time of the best focus for each pixel, creating a depth map.…”

“…Thus, SNNs are a popular choice for many systems using silicon retinas like the work of Orchard et al (2013), Orchard et al (2015), and Haessig et al (2017). Examples for event-based stereo vision applications applying networks with spiking neurons are Dikov et al (2017), Osswald et al (2017), Rebecq et al (2017), and Haessig et al (2019).…”

Data‐Driven Technology in Event‐Based Vision

“…Spiking neural networks (SNNs) [45] have been applied to various event-based fields, including low-level tasks such as optical flow estimation [46][47][48], high-level tasks such as object recognition [49,50] and classification [51], and tasks concerning the 3D structure of the scene [52,53] and robotic visual perception [54]. Benosman et al [16] used a spiking neural network that is theoretically similar to the classical Lucas-Kanade algorithm to estimate visual motion, exploiting the sparse high temporal resolution event data.…”