Realtime Time Synchronized Event-Based Stereo

Zhu, Alex Zihao; Chen, Yibo; Daniilidis, Kostas

doi:10.1007/978-3-030-01231-1_27

Cited by 69 publications

(43 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, brute-force space-sweeping using dedicated hardware (a GPU) has been proposed [191]. The method is based on ideas similar to [19], [112]: the correct depth manifests as "in focus" voxels of displaced events in the Disparity Space Image [19], [192].…”

Section: D Reconstruction Monocular and Stereomentioning

confidence: 99%

Event-Based Vision: A Survey

Gallego

Delbrück

Orchard

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

1,298

720

View full text Add to dashboard Cite

Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of µs), very high dynamic range (140 dB vs. 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world.

show abstract

Section: D Reconstruction Monocular and Stereomentioning

confidence: 99%

Event-Based Vision: A Survey

Gallego

Delbrück

Orchard

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

1,298

720

View full text Add to dashboard Cite

show abstract

“…Recently, there have been several works, such as [4,5,13,26,24], that have shown that optical flow, and other types of motion information, can be estimated from a spatiotemporal volume of events, by propagating the events along the optical flow direction, and attempting to minimize the motion blur in the event image. This concept of motion blur as a loss can be seen as an analogy to the photometric error in frames, as applied to events.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

Zhu

Yuan

Chaney

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

381

292

View full text Add to dashboard Cite

In this work, we propose a novel framework for unsupervised learning for event cameras that learns motion information from only the event stream. In particular, we propose an input representation of the events in the form of a discretized volume that maintains the temporal distribution of the events, which we pass through a neural network to predict the motion of the events. This motion is used to attempt to remove any motion blur in the event image. We then propose a loss function applied to the motion compensated event image that measures the motion blur in this image. We train two networks with this framework, one to predict optical flow, and one to predict egomotion and depths, and evaluate these networks on the Multi Vehicle Stereo Event Camera dataset, along with qualitative results from a variety of different scenes.

show abstract

“…Event cameras are thus sensors that can provide high-quality visual information even in challenging high-speed scenarios and high dynamic range environments, enabling new application domains for visionbased algorithms. Recently, these sensors have received great interest in various computer vision fields, ranging from computational photography [27,26,30,31] 1 to visual odometry [29,25,24,37,40,14] and depth prediction [15,25,22,36,38,33,40]. The survey in [7] gives a good overview of the applications for event cameras.…”

Section: Introductionmentioning

confidence: 99%

Learning Monocular Dense Depth from Events

Hidalgo-Carrió

Gehrig

Scaramuzza

2020

2020 International Conference on 3D Vision (3DV)

View full text Add to dashboard Cite

Event cameras are novel sensors that output brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. Compared to conventional image sensors, they offer significant advantages: high temporal resolution, high dynamic range, no motion blur, and much lower bandwidth. Recently, learning-based approaches have been applied to event-based data, thus unlocking their potential and making significant progress in a variety of tasks, such as monocular depth prediction. Most existing approaches use standard feed-forward architectures to generate network predictions, which do not leverage the temporal consistency presents in the event stream. We propose a recurrent architecture to solve this task and show significant improvement over standard feed-forward methods. In particular, our method generates dense depth predictions using a monocular setup, which has not been shown previously. We pretrain our model using a new dataset containing events and depth maps recorded in the CARLA simulator. We test our method on the Multi Vehicle Stereo Event Camera Dataset (MVSEC). Quantitative experiments show up to 50% improvement in average depth error with respect to previous event-based methods.

show abstract

Realtime Time Synchronized Event-Based Stereo

Cited by 69 publications

References 26 publications

Event-Based Vision: A Survey

Event-Based Vision: A Survey

Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

Learning Monocular Dense Depth from Events

Contact Info

Product

Resources

About