CED: Color Event Camera Dataset

Scheerlinck, Cedric; Rebecq, Henri; Stoffregen, Timo; Barnes, Nick; Mahony, Robert; Scaramuzza, Davide

doi:10.1109/cvprw.2019.00215

Cited by 64 publications

(45 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figs. 21, 22 and 23 present HDR reconstruction results on various publicly available datasets [5], [43], [45]. Further results are shown in the supplementary video which conveys these results in a better form than still images.…”

Section: D1 Results On Synthetic Event Datamentioning

confidence: 94%

“…Indoors [45] Outdoors [5] Night Drive [43] Night Drive [5] (a) Events (b) Frame (c) Reconstruction Fig. 23.…”

Section: D2 Additional Qualitative Results On Real Datamentioning

confidence: 99%

“…Video reconstruction under challenging lighting. First row: indoor sequence [45]. Second row: outdoor sequence from [5].…”

Section: D2 Additional Qualitative Results On Real Datamentioning

confidence: 99%

“…Color reconstruction from such events was first shown by [44], where a single color image was recovered from a large set of events using a method similar to [17]. Later, [45] adapted existing monochrome video reconstruction methods [4], [30] to color (a) Individual channels (b) [45] (upsampled, color) (c) [45] (details) (d) Grayscale (full res) (e) Ours (full res, color) (f) Ours (details) Fig. 10.…”

Section: Color Video Reconstructionmentioning

confidence: 99%

“…We now describe a simple method to perform color reconstruction from color event data at full resolution with our network, and then present qualitative results. Following [45], we reconstruct the four color channels independently at quarter resolution ( Fig. 10(a)), upsample with bicubic interpolation, and recombine them into a low-quality color image ( Fig.…”

Section: Color Video Reconstructionmentioning

confidence: 99%

See 4 more Smart Citations

High Speed and High Dynamic Range Video with an Event Camera

Rebecq

Ranftl

Koltun

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

389

339

View full text Add to dashboard Cite

Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our quantitative experiments show that our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. As an additional contribution, we demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code and a pre-trained model to enable further research.

show abstract

Section: D1 Results On Synthetic Event Datamentioning

confidence: 94%

“…Indoors [45] Outdoors [5] Night Drive [43] Night Drive [5] (a) Events (b) Frame (c) Reconstruction Fig. 23.…”

Section: D2 Additional Qualitative Results On Real Datamentioning

confidence: 99%

“…Video reconstruction under challenging lighting. First row: indoor sequence [45]. Second row: outdoor sequence from [5].…”

Section: D2 Additional Qualitative Results On Real Datamentioning

confidence: 99%

Section: Color Video Reconstructionmentioning

confidence: 99%

Section: Color Video Reconstructionmentioning

confidence: 99%

See 3 more Smart Citations

High Speed and High Dynamic Range Video with an Event Camera

Rebecq

Ranftl

Koltun

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

389

339

View full text Add to dashboard Cite

show abstract

A Differentiable Recurrent Surface for Asynchronous Event-Based Data

Cannici

Ciccone

Romanoni

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Reducing the Sim-to-Real Gap for Event Cameras

Stoffregen

Scheerlinck

Scaramuzza

et al. 2020

Lecture Notes in Computer Science

Self Cite

110

174

View full text Add to dashboard Cite

Event cameras are paradigm-shifting novel sensors that report asynchronous, per-pixel brightness changes called 'events' with unparalleled low latency. This makes them ideal for high speed, high dynamic range scenes where conventional cameras would fail. Recent work has demonstrated impressive results using Convolutional Neural Networks (CNNs) for video reconstruction and optic flow with events. We present strategies for improving training data for event based CNNs that result in 20-40% boost in performance of existing state-of-the-art (SOTA) video reconstruction networks retrained with our method, and up to 15% for optic flow networks. A challenge in evaluating event based video reconstruction is lack of quality ground truth images in existing datasets. To address this, we present a new High Quality Frames (HQF) dataset, containing events and ground truth frames from a DAVIS240C that are well-exposed and minimally motion-blurred. We evaluate our method on HQF + several existing major event camera datasets.Video, code and datasets: https://timostoff.github.io/20ecnn This paper has been accepted for publication at the European Conference on Computer Vision, 2020Reducing the Sim-to-Real Gap for Event Cameras 3 that provides perfectly aligned frames from an integrated Active Pixel Sensor (APS). HQF also contains a diverse range of motions and scene types, including slow motion and pauses that are challenging for event based video reconstruction. We quantitatively evaluate our method on two major event camera datasets: IJRR [23] and MVSEC [42], in addition to our HQF, demonstrating gains of 20-40 % for video reconstruction and up to 15 % for optic flow when we retrain existing SOTA networks.Contribution We present a method to generate synthetic training data that improves generalizability to real event data, guided by statistical analysis of existing datasets. We additionally propose a simple method for dynamic train-time noise augmentation that yields up to 10 % improvement for video reconstruction. Using our method, we retrain several network architectures from previously published works on video reconstruction [28,32] and optic flow [43, 44] from events. We are able to show significant improvements that persist over architectures and tasks. Thus, we believe our findings will provide invaluable insight for others who wish to train models on synthetic events for a variety of tasks. We provide a new comprehensive High Quality Frames dataset targeting ground truth image frames for video reconstruction evaluation. Finally, we provide our data generation code, training set, training code and our pretrained models, together with dozens of useful helper scripts for the analysis of event-based datasets to make this task easier for fellow researchers.In summary, our major contributions are:-A method for simulating training data that yields 20 %-40 and up to 15 % improvement for event based video reconstruction and optic flow CNNs. -Dynamic train-time event noise augmentation.-A novel High Quality Frames dataset.-Extensive ...

show abstract

CED: Color Event Camera Dataset

Cited by 64 publications

References 53 publications

High Speed and High Dynamic Range Video with an Event Camera

High Speed and High Dynamic Range Video with an Event Camera

A Differentiable Recurrent Surface for Asynchronous Event-Based Data

Reducing the Sim-to-Real Gap for Event Cameras

Contact Info

Product

Resources

About