Direct face detection and video reconstruction from event cameras

Barua, Souptik; Miyatani, Yoshitaka; Veeraraghavan, Ashok

doi:10.1109/wacv.2016.7477561

Cited by 103 publications

(91 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Naive integration of the event stream leads to very fast degradation of image quality due to accumulating noise. As a remedy, earlier works have proposed hand-crafted image priors to constrain the problem [2], [3], [4], arXiv:1906.07165v1 [cs.CV] 15 Jun 2019 [5]. However, these priors make strong assumptions about the statistics of natural images, leading to unrealistic reconstructions and artifacts.…”

Section: Introductionmentioning

confidence: 99%

High Speed and High Dynamic Range Video with an Event Camera

Rebecq

Ranftl

Koltun

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

391

339

View full text Add to dashboard Cite

Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our quantitative experiments show that our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. As an additional contribution, we demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code and a pre-trained model to enable further research.

show abstract

Section: Introductionmentioning

confidence: 99%

High Speed and High Dynamic Range Video with an Event Camera

Rebecq

Ranftl

Koltun

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

391

339

View full text Add to dashboard Cite

show abstract

“…Analogous to early frame-based computer vision approaches, significant effort has been made in designing efficient spatiotemporal feature descriptors of the event stream. From this line of research, typical high-level applications are gesture recognition [31], object recognition [29,45,59] or face detection [6]. Low-level applications include optical flow prediction [8,9] and image reconstruction [5].…”

Section: Related Workmentioning

confidence: 99%

“…Candidate Representations We start out by identifying 12 distinct representations based on the event spike tensor (6). In particular, we select the measurement function (4) from three candidates: event polarity, event count, and normalized time stamp.…”

Section: Empirical Evaluationmentioning

confidence: 99%

End-to-End Learning of Representations for Asynchronous Event-Based Data

Gehrig

Loquercio

Derpanis

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

277

237

View full text Add to dashboard Cite

Event cameras are vision sensors that record asynchronous streams of per-pixel brightness changes, referred to as "events". They have appealing advantages over frame-based cameras for computer vision, including high temporal resolution, high dynamic range, and no motion blur. Due to the sparse, non-uniform spatiotemporal layout of the event signal, pattern recognition algorithms typically aggregate events into a grid-based representation and subsequently process it by a standard vision pipeline, e.g., Convolutional Neural Network (CNN). In this work, we introduce a general framework to convert event streams into grid-based representations through a sequence of differentiable operations. Our framework comes with two main advantages: (i) allows learning the input event representation together with the task dedicated network in an end-to-end manner, and (ii) lays out a taxonomy that unifies the majority of extant event representations in the literature and identifies novel ones. Empirically, we show that our approach to learning the event representation end-to-end yields an improvement of approximately 12% on optical flow estimation and object recognition over state-of-the-art methods.

show abstract

“…To visually unveil the color information contained in color events, we evaluate and compare three state-of-the-art event-based image reconstruction methods [22,24,50] on our Color Event Camera Dataset. Image reconstruction is an active field of event-based vision research [6,[19][20][21][22][23][24]50] that allows visualisation of the event stream, and enables application of decades of computer vision research and expertise on event data, which in its raw form is inaccessible to powerful tools such as convolutional neural networks. Further, event reconstructed images have the potential to retain desirable qualities of event cameras, such as high dynamic range, high temporal resolution and immunity to motion blur.…”

Section: Introductionmentioning

confidence: 99%

CED: Color Event Camera Dataset

Scheerlinck

Rebecq

Stoffregen

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

Event cameras are novel, bio-inspired visual sensors, whose pixels output asynchronous and independent timestamped spikes at local intensity changes, called 'events'. Event cameras offer advantages over conventional framebased cameras in terms of latency, high dynamic range (HDR) and temporal resolution. Until recently, event cameras have been limited to outputting events in the intensity channel, however, recent advances have resulted in the development of color event cameras, such as the Color-DAVIS346. In this work, we present and release the first Color Event Camera Dataset (CED), containing 50 minutes of footage with both color frames and events. CED features a wide variety of indoor and outdoor scenes, which we hope will help drive forward event-based vision research. We also present an extension of the event camera simulator ESIM [1] that enables simulation of color events. Finally, we present an evaluation of three state-of-the-art image reconstruction methods that can be used to convert the Color-DAVIS346 into a continuous-time, HDR, color video camera to visualise the event stream, and for use in downstream vision applications.

show abstract

Direct face detection and video reconstruction from event cameras

Cited by 103 publications

References 29 publications

High Speed and High Dynamic Range Video with an Event Camera

High Speed and High Dynamic Range Video with an Event Camera

End-to-End Learning of Representations for Asynchronous Event-Based Data

CED: Color Event Camera Dataset

Contact Info

Product

Resources

About