Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera

Abstract

Event-based cameras can measure intensity changes (called 'events') with microsecond accuracy under highspeed motion and challenging lighting conditions. With the active pixel sensor (APS), the event camera allows simultaneous output of the intensity frames. However, the output images are captured at a relatively low frame-rate and often suffer from motion blur. A blurry image can be regarded as the integral of a sequence of latent images, while the events indicate the changes between the latent images. Therefore, we are able to model the blur-generation process by associating event data to a latent image. In this paper, we propose a simple and effective approach, the Event-based Double Integral (EDI) model, to reconstruct a high framerate, sharp video from a single blurry frame and its event data. The video generation is based on solving a simple non-convex optimization problem in a single scalar variable. Experimental results on both synthetic and real images demonstrate the superiority of our EDI model and optimization method in comparison to the state-of-the-art.

Event-Driven Video Frame Synthesis

“…A motion blurred image (a) and the events during exposure time (b) are used to reconstruct a high framerate video. Compare to (c) EDI[32], our results (d) preserves spatial features with less noise.…”

“…Our framework includes two key steps, i.e., DMR and RD. Our DMR is free of training and is capable to unify different fusion settings between the two sensing modalities, which was not considered in previous work such as [32,36]. We show in real data that our DMR performs better than existing algorithms.…”

“…Converting event streams (binary) to multiple-valued intensity frames is a challenging task, yet has been shown beneficial to downstream visual tasks [34]. Existing strategies for image reconstruction include dictionary learning [3], manifold regularization [30], optical flow [2], exponential integration [32,36], conditional Generative Adversarial Networks (GAN) [40] and recurrent neural network [34]. Compared to existing algorithms, our work is the first, to the best of our knowledge, to unify different temporal frame synthesis settings, including interpolation, extrapolation (prediction) and motion deblur (reconstructing a video from a motionblurred image).…”

“…(a) Blurry images (data from[32]) (b) w/ events during exposure time (c) EDI[32] (d) Ours (DMR) Motion deblur. A motion blurred image (a) and the events during exposure time (b) are used to reconstruct a high framerate video.…”

CED: Color Event Camera Dataset

“…Since their introduction, event cameras have spawned a flurry of research. They have been used in feature detection and tracking [3][4][5][6], depth estimation [7][8][9][10], stereo [11][12][13][14], optical flow [15][16][17][18], image reconstruction [19][20][21][22][23][24][25], localization [26][27][28][29], SLAM [30][31][32], visualinertial odometry [33][34][35][36], pattern recognition [37][38][39][40], and more. In response to the growing needs of the community, several important event-based vision datasets have been released, directed at popular topics such as SLAM [28], optical flow [41,42] and recognition [37,43].…”