Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms

Abstract

Any visual sensor, whether artificial or biological, maps the 3D-world on a 2D-representation. The missing dimension is depth and most species use stereo vision to recover it. Stereo vision implies multiple perspectives and matching, hence it obtains depth from a pair of images. Algorithms for stereo vision are also used prosperously in robotics. Although, biological systems seem to compute disparities effortless, artificial methods suffer from high energy demands and latency. The crucial part is the correspondence problem; finding the matching points of two images. The development of event-based cameras, inspired by the retina, enables the exploitation of an additional physical constraint—time. Due to their asynchronous course of operation, considering the precise occurrence of spikes, Spiking Neural Networks take advantage of this constraint. In this work, we investigate sensors and algorithms for event-based stereo vision leading to more biologically plausible robots. Hereby, we focus mainly on binocular stereo vision.

Event-Based Vision: A Survey

“…Over the next decade the neuromorphic community developed a series of silicon retinas. These developments are summarized in [36], [38], [42], [43].…”

Event-Based Visual Inertial Odometry

“…For example, [26] combines epipolar constraints, temporal inconsistency, motion inconsistency and photometric error (available only from grayscale events given by ATIS cameras [27]) into an objective function to compute the best matches. Other works, such as [28], [29], [30], extend cooperative stereo [31] to the case of event cameras [32]. These methods work well with static cameras in uncluttered scenes, so that event matches are easy to find among few moving objects.…”

Data_Sheet_1.PDF

“…This would allow us to use a smaller feature space compared to the resolution of the vision sensor, and increase robustness to noise in the vision sensors. As discussed in Steffen et al (2019), although there are many methods for event-based depth estimation, the lack of a comprehensive dataset or a standard testbed makes it difficult to compare them. Yet, some event-based datasets for stereo vision have been recently released (Andreopoulos et al, 2018;Zhu et al, 2018).…”

“…Typical applications in robotics that can benefit from stereo vision include navigation in unknown environments, object manipulation, and grasping. However, current machine-vision approaches still lag behind their biological counterpart mainly in terms of bandwidth and power consumption (Tippetts et al, 2016;Steffen et al, 2019). Classical methods are based on frame-based vision sensors.…”

“…Biological systems, on the other hand, seem to efficiently solve the stereo-matching problem by using space-variant and asynchronous space-time sampling (Steffen et al, 2019). Space-variant resolution refers to a non-uniform distribution of retinal photoreceptors, with higher density in the center (fovea) and a decreasing density toward the periphery.…”