Focus Is All You Need: Loss Functions for Event-Based Vision

Abstract

θ I(x; θ) Parameters Warped Events Focus score Warping along point trajectories X [pix] Y [pix] time [s] X [pix] Y [pix] Measure event alignment Input Events θFigure 1: Motion Compensation Framework. Events in a space-time window are warped according to point trajectories described by motion parameters θ, resulting in an image of warped events (IWE) I(x; θ). Then, a focus loss function of I measures how well events are aligned along the point trajectories. This work proposes multiple focus loss functions for event alignment (last block in the figure and Table 1) for tasks such as rotational motion, depth and optical flow estimation. AbstractEvent cameras are novel vision sensors that output pixellevel brightness changes ("events") instead of traditional video frames. These asynchronous sensors offer several advantages over traditional cameras, such as, high temporal resolution, very high dynamic range, and no motion blur. To unlock the potential of such sensors, motion compensation methods have been recently proposed. We present a collection and taxonomy of twenty two objective functions to analyze event alignment in motion compensation approaches (Fig. 1). We call them focus loss functions since they have strong connections with functions used in traditional shapefrom-focus applications. The proposed loss functions allow bringing mature computer vision tools to the realm of event cameras. We compare the accuracy and runtime performance of all loss functions on a publicly available dataset, and conclude that the variance, the gradient and the Laplacian magnitudes are among the best loss functions. The applicability of the loss functions is shown on multiple tasks: rotational motion, depth and optical flow estimation. The proposed focus loss functions allow to unlock the outstanding properties of event cameras. †

Event-Based Vision: A Survey

“…Interpolated voxel-grid (240 × 180 × 10 voxels), colored according to polarity, from dark (negative) to bright (positive). Motion-compensated event image [82] (sharp edges obtained by event accumulation are darker than pixels with no events, in white). Reconstructed intensity image by [8].…”

“…The idea of motion compensation is that, as an edge moves on the image plane, it triggers events on the pixels it traverses; the motion of the edge can be estimated by warping the events to a reference time and maximizing their alignment, producing a sharp image (i.e., histogram) of warped events (IWE) [112]. Hence, this representation (IWE) suggests a criterion to measure how well events fit a candidate motion: the sharper the edges produced by warping events, the better the fit [82]. Moreover, the resulting motion-compensated images have an intuitive meaning (i.e., the edge patterns causing the events) and provide a more familiar representation of visual information than the events.…”

Event-Based Visual Inertial Odometry

“…The recovery of intensity information and depth regularization make the method computationally intensive, thus requiring dedicated hardware (GPU) for real-time operation. In contrast, [14] proposes a geometric approach based on the semi-dense mapping technique in [33] (focusing events [41]) and an image alignment tracker that works on event images. It does not need to recover absolute intensity and runs in real time on a CPU.…”

Event-Based Eccentric Motion Detection Exploiting Time Difference Encoding

“…We propose that these models can also be used to direct attention toward moving objects within a scene. Recent studies have developed event-based motion detection for optical flow estimation both relying on conventional processing architectures (Benosman et al, 2012(Benosman et al, , 2014Gallego et al, 2018Gallego et al, , 2019Mitrokhin et al, 2018) and unconventional neuromorphic processing architectures (Giulioni et al, 2016;Haessig et al, 2018;Milde et al, 2018). Even though the former mechanisms, which leverage standard processing capabilities, show real-time optic flow estimation with very high accuracy, they are not suited for spiking neural networks and neuromorphic processors.…”

“…This is due to the way information is represented, using real values in these algorithms. Additionally, the power consumption and computational complexity in Gallego et al (2018Gallego et al ( , 2019 is too high for constrained robotic tasks. The neuromorphic approaches on the other hand can naturally interact with spiking networks implemented on low-power neuromorphic processing architectures as information is encoded using events.…”