Event-Based Stereo Depth Estimation Using Belief Propagation

Abstract

Compared to standard frame-based cameras, biologically-inspired event-based sensors capture visual information with low latency and minimal redundancy. These event-based sensors are also far less prone to motion blur than traditional cameras, and still operate effectively in high dynamic range scenes. However, classical framed-based algorithms are not typically suitable for these event-based data and new processing algorithms are required. This paper focuses on the problem of depth estimation from a stereo pair of event-based sensors. A fully event-based stereo depth estimation algorithm which relies on message passing is proposed. The algorithm not only considers the properties of a single event but also uses a Markov Random Field (MRF) to consider the constraints between the nearby events, such as disparity uniqueness and depth continuity. The method is tested on five different scenes and compared to other state-of-art event-based stereo matching methods. The results show that the method detects more stereo matches than other methods, with each match having a higher accuracy. The method can operate in an event-driven manner where depths are reported for individual events as they are received, or the network can be queried at any time to generate a sparse depth frame which represents the current state of the network.

Realtime Time Synchronized Event-Based Stereo

“…With further optimizations and an implementation in raw CUDA or OpenCL, we expect this time to reduce further. This corresponds to a runtime of around 2.7µs per event, compared to the 0.65-2ms reported in [17]. However, it should be noted that the competing methods were implemented in MATLAB on CPU, and would almost certainly see speed improvements if ported to other languages/devices.…”

“…In addition, several works have applied smoothing based regularizations to constrain ambiguous regions, which have seen great success in frame based stereo. Piatkowska et al [11,12], have applied cooperative stereo methods [8] in an asynchronous fashion, while Xie et al [17,18] have adapted belief propagation [1] and semiglobal matching [4], respectively, to similar effect. These regularizations have shown significant improvements over the prior state of the art.…”

“…A number of event based stereo methods have approached these problem with asynchronous methods (e.g. [11,15,17]), which process each event independently. However, these methods must either forego the information provided by the spatial neighborhood around each event, or use fine tuned temporal windows to once again ensure time synchronization, as there are no guarantees that neighboring events were generated at a similar time.…”

Focus Is All You Need: Loss Functions for Event-Based Vision

“…where, using (16) and result (53), the integrand becomes ∂ ∂θ Var(x; I) (16) ≈ 2I(x) ∂I(x) ∂θ * G σ (x) − 2(I(x) * G σ (x)) ∂I(x) ∂θ * G σ (x) .…”

“…Event cameras, such as the Dynamic Vision Sensor (DVS) [1] posses outstanding properties compared to traditional cameras: very high dynamic range (140 dB vs. 60 dB), high temporal resolution (in the order of µs), and do not suffer from motion blur. Hence, event cameras have a large potential to tackle challenging scenarios for standard cameras (such as high speed and high dynamic range) in tracking [2][3][4][5][6][7][8][9], depth estimation [10][11][12][13][14][15][16][17][18][19], Simultaneous Localization and Mapping [20][21][22][23][24][25][26][27], and recognition [28][29][30][31][32], among other applications. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential.…”