Event Collapse in Contrast Maximization Frameworks

Abstract

Contrast maximization (CMax) is a framework that provides state-of-the-art results on several event-based computer vision tasks, such as ego-motion or optical flow estimation. However, it may suffer from a problem called event collapse, which is an undesired solution where events are warped into too few pixels. As prior works have largely ignored the issue or proposed workarounds, it is imperative to analyze this phenomenon in detail. Our work demonstrates event collapse in its simplest form and proposes collapse metrics by using first principles of space–time deformation based on differential geometry and physics. We experimentally show on publicly available datasets that the proposed metrics mitigate event collapse and do not harm well-posed warps. To the best of our knowledge, regularizers based on the proposed metrics are the only effective solution against event collapse in the experimental settings considered, compared with other methods. We hope that this work inspires further research to tackle more complex warp models.

Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow

“…This allowed us to focus on reformulating the image reconstruction part as a linear inverse problem and propose efficient, explainable solvers. The experiments so far used the flow estimated by methods designed for low degrees-of-freedom (DOF) motions [26], [60], [61], [62], which are often a good approximation and produce very accurate results if the true motion satisfies their assumptions [53], [63]. More complex scenes may require dense optical flow estimation, as given by [27], [28], [31] (i.e., higher DOFs).…”

Secrets of Event-Based Optical Flow

“…Zhu et al [58] report that the contrast objective (variance) overfits to the events. This is in part because the warp (4) can describe very complex flow fields, which can push the events to accumulate in few pixels [42]. To mitigate overfitting, we reduce the complexity of the flow field by dividing the image plane into a tile of non-overlapping patches, defining a flow vector at the center of each patch and interpolating the flow on all other pixels (we show the tiles in Sec.…”

“…The intuitive interpretation is to estimate the motion by recovering the sharp (motion-compensated) image of edge patterns that caused the events. Preliminary work on applying CM to estimate optical flow has reported a problem of overfitting to the data, producing undesired flows that warp events to few pixels or lines [58] (i.e., event collapse [42]). This issue has been tackled by changing the objective function, from contrast to the energy of an average timestamp image [21,34,58], but this loss is not straightforward to interpret and makes training difficult to converge [13].…”

Multi-Event-Camera Depth Estimation and Outlier Rejection by Refocused Events Fusion

“…The effectiveness of EVO is largely due to its mapping module, Event-based Multi-View Stereo (EMVS) [10], which enables 3D reconstruction without the need to recover image intensity, without having to explicitly solve for data association between events, and without the need of a GPU (it is fast on a standard CPU -e.g., speed of 1.20 Mev/s/core [10]). Additionally, EMVS admits an interpretation in terms of event refocusing or event alignment (contrast maximization) [16], which is the state of the art framework to tackle other vision problems [17]- [25]. Our goal is to extend EMVS to the multi-camera setting (i.e., two or more event cameras in a multi-view configuration sharing a common clock), and in particular to the stereo setting, in order to benefit from these advantages and connections (Fig.…”