Secrets of Event-Based Optical Flow

Shiba, Shintaro; Aoki, Yoshimitsu; Gallego, Guillermo

doi:10.1007/978-3-031-19797-0_36

Cited by 53 publications

(48 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experiments so far used the flow estimated by methods designed for low degrees-of-freedom (DOF) motions [26], [60], [61], [62], which are often a good approximation and produce very accurate results if the true motion satisfies their assumptions [53], [63]. More complex scenes may require dense optical flow estimation, as given by [27], [28], [31] (i.e., higher DOFs).…”

Section: Combination With State-of-the-art Dense Optical Flow Methodsmentioning

confidence: 99%

“…Figure 18 shows the result of our IWE method when combined with two state-of-the-art event-based dense optical flow methods: E-RAFT [28] (top row) and Multi- reference Contrast Maximization [27] (MCM, bottom row). E-RAFT is a supervised deep learning method based on RAFT [64], and it uses the events in a time window of 100 ms. MCM is a model-based method that uses a fixed number of events (500k in the example).…”

Section: Combination With State-of-the-art Dense Optical Flow Methodsmentioning

confidence: 99%

“…These model-based approaches suffer from artifacts such as ghosting effects and bleeding edges. Compared to [18], our method assumes optical flow is obtained first, for example via contrast maximization [26], [27] or E-RAFT [28], and then it recovers brightness. The goal is to highlight and exploit the asymmetries in the combined problem of recovering flow and brightness, which are entangled variables in the event stream.…”

Section: Related Workmentioning

confidence: 99%

“…For comparison, RNNs using optimized implementations take: 0.2448 s (E2VID), 0.2839 s (ECNN) and 0.4059 s (BTEB) on the same CPU. The time to covert the events into a voxel grid to feed to the RNNs is 0.01 s. Likewise, the unoptimized MCM method E-RAFT [28] MCM [27] (a) IWE (events) (b) Ours (CNN) Fig. 18: Image reconstruction in combination with off-theshelf dense optical flow methods [28] and [27].…”

Section: Computational Performancementioning

confidence: 99%

See 3 more Smart Citations

Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow

Zhang

Yezzi

Gallego

2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Event cameras are novel bio-inspired sensors that measure per-pixel brightness differences asynchronously. Recovering brightness from events is appealing since the reconstructed images inherit the high dynamic range (HDR) and high-speed properties of events; hence they can be used in many robotic vision applications and to generate slow-motion HDR videos. However, state-of-the-art methods tackle this problem by training an event-to-image Recurrent Neural Network (RNN), which lacks explainability and is difficult to tune. In this work we show, for the first time, how tackling the combined problem of motion and brightness estimation leads us to formulate event-based image reconstruction as a linear inverse problem that can be solved without training an image reconstruction RNN. Instead, classical and learning-based regularizers are used to solve the problem and remove artifacts from the reconstructed images. The experiments show that the proposed approach generates images with visual quality on par with state-of-the-art methods despite only using data from a short time interval. State-of-the-art results are achieved using an image denoising Convolutional Neural Network (CNN) as the regularization function. The proposed regularized formulation and solvers have a unifying character because they can be applied also to reconstruct brightness from the second derivative. Additionally, the formulation is attractive because it can be naturally combined with super-resolution, motion-segmentation and color demosaicing. Code is available at https://github.com/tub-rip/event based image rec inverse problem

show abstract

Section: Combination With State-of-the-art Dense Optical Flow Methodsmentioning

confidence: 99%

Section: Combination With State-of-the-art Dense Optical Flow Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Computational Performancementioning

confidence: 99%

See 2 more Smart Citations

Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow

Zhang

Yezzi

Gallego

2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Studying how the motion of a stereo event camera affects 3D visual perception could improve understanding of binocular vision and help design algorithms as efficient and robust as biological systems. Moreover, the outstanding properties of event cameras, such as high dynamic range (HDR), high temporal resolution (≈ µs) and low power consumption, offer potential to tackle scenarios that are challenging for standard cameras (high speed and/or HDR) [1,3,6,7,15,16,18]. This extended abstract is based on our recent paper on multi-event camera depth estimation [8].…”

Section: Introductionmentioning

confidence: 99%

Event-based Stereo Depth Estimation from Ego-motion using Ray Density Fusion

Ghosh¹,

Gallego²

2022

Preprint

View full text Add to dashboard Cite

Event cameras are bio-inspired sensors that mimic the human retina by responding to brightness changes in the scene. They generate asynchronous spike-based outputs at microsecond resolution, providing advantages over traditional cameras like high dynamic range, low motion blur and power efficiency. Most event-based stereo methods attempt to exploit the high temporal resolution of the camera and the simultaneity of events across cameras to establish matches and estimate depth. By contrast, this work investigates how to estimate depth from stereo event cameras without explicit data association by fusing back-projected ray densities, and demonstrates its effectiveness on head-mounted camera data, which is recorded in an egocentric fashion. Code and video are available at https://github.com/tub-rip/dvs mcemvs

show abstract

Multi‐Event‐Camera Depth Estimation and Outlier Rejection by Refocused Events Fusion

Ghosh

Gallego

2022

Advanced Intelligent Systems

View full text Add to dashboard Cite

Event cameras are bio‐inspired sensors that offer advantages over traditional cameras. They operate asynchronously, sampling the scene at microsecond resolution and producing a stream of brightness changes. This unconventional output has sparked novel computer vision methods to unlock the camera's potential. Here, the problem of event‐based stereo 3D reconstruction for SLAM is considered. Most event‐based stereo methods attempt to exploit the high temporal resolution of the camera and the simultaneity of events across cameras to establish matches and estimate depth. By contrast, this work investigates how to estimate depth without explicit data association by fusing disparity space images (DSIs) originated in efficient monocular methods. Fusion theory is developed and applied to design multi‐camera 3D reconstruction algorithms that produce state‐of‐the‐art results, as confirmed by comparisons with four baseline methods and tests on a variety of available datasets.

show abstract

Secrets of Event-Based Optical Flow

Cited by 53 publications

References 54 publications

Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow

Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow

Event-based Stereo Depth Estimation from Ego-motion using Ray Density Fusion

Multi‐Event‐Camera Depth Estimation and Outlier Rejection by Refocused Events Fusion

Contact Info

Product

Resources

About