Event-Based Motion Segmentation by Motion Compensation

Stoffregen, Timo; Gallego, Guillermo; Drummond, Tom; Kleeman, Lindsay; Scaramuzza, Davide

doi:10.1109/iccv.2019.00734

Cited by 147 publications

(133 citation statements)

References 38 publications

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…As a consequence, custom algorithms need to be specifically tailored to leverage event data. Such specialized algorithms have demonstrated impressive performance in applications ranging from low-level vision tasks, such as visual odometry [18,40,55,39,43], feature tracking [20,54,15] and optical flow [5,3,49,57,48], to high-level tasks such as object classification [36,21,47] and gesture recognition [2].…”

Section: Downstream Applicationsmentioning

confidence: 99%

Events-To-Video: Bringing Modern Computer Vision to Event Cameras

Rebecq

Ranftl

Koltun

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

306

227

View full text Add to dashboard Cite

Event cameras are novel sensors that report brightness changes in the form of asynchronous "events" instead of intensity frames. They have significant advantages over conventional cameras: high temporal resolution, high dynamic range, and no motion blur. Since the output of event cameras is fundamentally different from conventional cameras, it is commonly accepted that they require the development of specialized algorithms to accommodate the particular nature of events. In this work, we take a different view and propose to apply existing, mature computer vision techniques to videos reconstructed from event data. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. Our experiments show that our approach surpasses state-of-the-art reconstruction methods by a large margin (> 20%) in terms of image quality. We further apply off-the-shelf computer vision algorithms to videos reconstructed from event data on tasks such as object classification and visual-inertial odometry, and show that this strategy consistently outperforms algorithms that were specifically designed for event data. We believe that our approach opens the door to bringing the outstanding properties of event cameras to an entirely new range of tasks. A video of the experiments is available at https://youtu.be/IdYrC4cUO0I

show abstract

Section: Downstream Applicationsmentioning

confidence: 99%

Events-To-Video: Bringing Modern Computer Vision to Event Cameras

Rebecq

Ranftl

Koltun

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

306

227

View full text Add to dashboard Cite

show abstract

“…(formally, the same formula as (40), but with the centered IWE playing the role of the IWE in (40)) with…”

Section: K Analytical Derivatives Of Focus Loss Functionsmentioning

confidence: 99%

“…Motion compensation approaches [15,19,[33][34][35][36][37][38][39][40][41][42] have been recently introduced for processing the visual information acquired by event cameras. They have proven successful for the estimation of motion (optical flow) [34-36, 38, 39], camera motion [33,35], depth (3D reconstruction) [15,19,38,39] as well as segmentation [36,40,41]. The main idea of such methods consists of searching for point trajectories on the image plane that maximize event alignment [33,35] (Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Focus Is All You Need: Loss Functions for Event-Based Vision

Gallego

Gehrig

Scaramuzza

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

120

116

View full text Add to dashboard Cite

θ I(x; θ) Parameters Warped Events Focus score Warping along point trajectories X [pix] Y [pix] time [s] X [pix] Y [pix] Measure event alignment Input Events θFigure 1: Motion Compensation Framework. Events in a space-time window are warped according to point trajectories described by motion parameters θ, resulting in an image of warped events (IWE) I(x; θ). Then, a focus loss function of I measures how well events are aligned along the point trajectories. This work proposes multiple focus loss functions for event alignment (last block in the figure and Table 1) for tasks such as rotational motion, depth and optical flow estimation. AbstractEvent cameras are novel vision sensors that output pixellevel brightness changes ("events") instead of traditional video frames. These asynchronous sensors offer several advantages over traditional cameras, such as, high temporal resolution, very high dynamic range, and no motion blur. To unlock the potential of such sensors, motion compensation methods have been recently proposed. We present a collection and taxonomy of twenty two objective functions to analyze event alignment in motion compensation approaches (Fig. 1). We call them focus loss functions since they have strong connections with functions used in traditional shapefrom-focus applications. The proposed loss functions allow bringing mature computer vision tools to the realm of event cameras. We compare the accuracy and runtime performance of all loss functions on a publicly available dataset, and conclude that the variance, the gradient and the Laplacian magnitudes are among the best loss functions. The applicability of the loss functions is shown on multiple tasks: rotational motion, depth and optical flow estimation. The proposed focus loss functions allow to unlock the outstanding properties of event cameras. †

show abstract

“…The optical flow of the events was used in [15] to segment the scene by clustering motion-compensated images into objects according to their velocity. Motion compensation was also used in [16] to provide a cluster association for each event while estimating the motion parameters of the objects through an optimization process. Recently, the work in [17] developed the first approach for event-based independent motion detection using Neural Networks to estimate the camera egomotion and segment both depth and pixel motion.…”

Section: State Of the Artmentioning

confidence: 99%

Asynchronous event-based clustering and tracking for intrusion monitoring in UAS

Eguíluz

Dios

Ollero

2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Automatic surveillance and monitoring using Unmanned Aerial Systems (UAS) require the development of perception systems that robustly work under different illumination conditions. Event cameras are neuromorphic sensors that capture the illumination changes in the scene with very low latency and high dynamic range. Although recent advances in eventbased vision have explored the use of event cameras onboard UAS, most techniques group events in frames and, therefore, do not fully exploit the sequential and asynchronous nature of the event stream. This paper proposes a fully asynchronous scheme for intruder monitoring using UAS. It employs efficient event clustering and feature tracking modules and includes a sampling mechanism to cope with the computational cost of event-by-event processing adapting to on-board hardware computational constraints. The proposed scheme was tested on a real multirotor in challenging scenarios showing significant accuracy and robustness to lighting conditions.

show abstract

Event-Based Motion Segmentation by Motion Compensation

Cited by 147 publications

References 38 publications

Events-To-Video: Bringing Modern Computer Vision to Event Cameras

Events-To-Video: Bringing Modern Computer Vision to Event Cameras

Focus Is All You Need: Loss Functions for Event-Based Vision

Asynchronous event-based clustering and tracking for intrusion monitoring in UAS

Contact Info

Product

Resources

About