Reducing the Sim-to-Real Gap for Event Cameras

Abstract

Event cameras are paradigm-shifting novel sensors that report asynchronous, per-pixel brightness changes called 'events' with unparalleled low latency. This makes them ideal for high speed, high dynamic range scenes where conventional cameras would fail. Recent work has demonstrated impressive results using Convolutional Neural Networks (CNNs) for video reconstruction and optic flow with events. We present strategies for improving training data for event based CNNs that result in 20-40% boost in performance of existing state-of-the-art (SOTA) video reconstruction networks retrained with our method, and up to 15% for optic flow networks. A challenge in evaluating event based video reconstruction is lack of quality ground truth images in existing datasets. To address this, we present a new High Quality Frames (HQF) dataset, containing events and ground truth frames from a DAVIS240C that are well-exposed and minimally motion-blurred. We evaluate our method on HQF + several existing major event camera datasets.Video, code and datasets: https://timostoff.github.io/20ecnn This paper has been accepted for publication at the European Conference on Computer Vision, 2020Reducing the Sim-to-Real Gap for Event Cameras 3 that provides perfectly aligned frames from an integrated Active Pixel Sensor (APS). HQF also contains a diverse range of motions and scene types, including slow motion and pauses that are challenging for event based video reconstruction. We quantitatively evaluate our method on two major event camera datasets: IJRR [23] and MVSEC [42], in addition to our HQF, demonstrating gains of 20-40 % for video reconstruction and up to 15 % for optic flow when we retrain existing SOTA networks.Contribution We present a method to generate synthetic training data that improves generalizability to real event data, guided by statistical analysis of existing datasets. We additionally propose a simple method for dynamic train-time noise augmentation that yields up to 10 % improvement for video reconstruction. Using our method, we retrain several network architectures from previously published works on video reconstruction [28,32] and optic flow [43, 44] from events. We are able to show significant improvements that persist over architectures and tasks. Thus, we believe our findings will provide invaluable insight for others who wish to train models on synthetic events for a variety of tasks. We provide a new comprehensive High Quality Frames dataset targeting ground truth image frames for video reconstruction evaluation. Finally, we provide our data generation code, training set, training code and our pretrained models, together with dozens of useful helper scripts for the analysis of event-based datasets to make this task easier for fellow researchers.In summary, our major contributions are:-A method for simulating training data that yields 20 %-40 and up to 15 % improvement for event based video reconstruction and optic flow CNNs. -Dynamic train-time event noise augmentation.-A novel High Quality Frames dataset.-Extensive ...

Experimental Comparison between Event and Global Shutter Cameras

“…More recently, Ref. [ 45 ] outperformed E2VID on certain event datasets by training the neural network on augmented simulated data with a wider range of event rates and contrast thresholds. We did not use [ 45 ] as we found it after we had processed our experimental data using E2VID.…”

N-ROD: a Neuromorphic Dataset for Synthetic-to-Real Domain Adaptation

“…Synth-to-Real vs Sim-to-Real. Using simulation on one side and real event data on the other indirectly introduces the sim-to-real gap studied in [11,30]. In order to understand how much this second domain shift affects performance, we compare our results with the ones which would been obtained by using simulation even to extract events from real (target) image.…”

“…Thus, the N-ROD setting is different from the one in [23], where event simulation is applied on both the source and target domains. Simultaneously, using simulation on one side and real event data on the other, N-ROD indirectly introduces the sim-to-real gap studied in [11,30]. The result is a double domain-shift, which combines both the synth-to-real and the sim-to-real shifts.…”

“…Very new is UDA in event-based data. Its effectiveness in tackling the gap between real events and simulated ones was recently shown in [23], which deviates from previous approaches acting on simulation parameters [30] and data augmentation [11]. Moreover, the authors of [23] show that, when RGB is the primary modality, adding simulated events obtained from both synthetic source images and real target ones can be used as a tool in UDA to mitigate the intrinsic synth-to-real shift on images.…”

“…A common option to mitigate this issue is to use data simulation as an alternative to direct training on real data, as it provides a way to re-purpose existing RGB datasets in the event domain. However, the simulations do not always match the data distribution coming from a real sensor [11,30]. As a result, training on simulated data could lead to sub-optimal performance.…”