Video Synthesis from Intensity and Event Frames

Abstract

Event cameras, neuromorphic devices that naturally respond to brightness changes, have multiple advantages with respect to traditional cameras. However, the difficulty of applying traditional computer vision algorithms on event data limits their usability. Therefore, in this paper we investigate the use of a deep learning-based architecture that combines an initial grayscale frame and a series of event data to estimate the following intensity frames. In particular, a fully-convolutional encoder-decoder network is employed and evaluated for the frame synthesis task on an automotive event-based dataset. Performance obtained with pixel-wise metrics confirms the quality of the images synthesized by the proposed architecture.

Combining Events and Frames Using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction

“…This is why for more complex tasks such as image reconstruction or monocular depth, state-of-the-art methods use a data-driven approach [7], [6], [19], [20], [21].Of these, many rely on recurrent architectures which can leverage long time windows of events for improved prediction [21], [7]. Although there exist many purely event-based learning methods, few address the fusion of images and events [9], [10], [22]. These approaches fuse both modalities by synchronizing and concatenating both inputs and passing them to a standard feed-forward network [9], [10], [22].…”

“…They have identical residuals and decoders but instead of recurrent state combination operators, they feature recurrent convLSTM encoders at each level. While E only receives voxel grids as input, I receives only gray-scale frames and E+I receives stacks of voxel grids and frames, similar to [9]. For E+I when a new voxel grid arrives, we stack it with a copy of the last seen image.…”

“…Although there exist many purely event-based learning methods, few address the fusion of images and events [9], [10], [22]. These approaches fuse both modalities by synchronizing and concatenating both inputs and passing them to a standard feed-forward network [9], [10], [22]. While this strategy improves over passing each input individually, it discards the asynchronous nature and high temporal resolution of the events through stacking and synchronization.…”

“…1-C in the appendix. While E only receives voxel grids as input, I receives only gray-scale frames and E+I receives stacks of voxel grids and frames, similar to [9]. For E+I when a new voxel grid arrives, we stack it with a copy of the last seen image.…”

“…By contrast, learning-based methods have leveraged large datasets to generate more accurate predictions. However, current learning-based methods for events are limited in that they group events and frames into synchronized stacks which are passed to feed-forward neural networks [9], [10]. Not only does this strategy sacrifice the asynchronicity and high temporal resolution of events, but it also limits the temporal context by using simple feed-forward networks instead of RNNs.…”