An Asynchronous Kalman Filter for Hybrid Event Cameras

Abstract

Event cameras are ideally suited to capture High Dynamic Range (HDR) visual information without blur but provide poor imaging capability for static or slowly varying scenes. Conversely, conventional image sensors measure absolute intensity of slowly changing scenes effectively but do poorly on HDR or quickly changing scenes. In this paper, we present an asynchronous linear filter architecture, fusing event and frame camera data, for HDR video reconstruction and spatial convolution that exploits the advantages of both sensor modalities. The key idea is the introduction of a state that directly encodes the integrated or convolved image information and that is updated asynchronously as each event or each frame arrives from the camera. The state can be read-off as-often-as and whenever required to feed into subsequent vision modules for real-time robotic systems. Our experimental results are evaluated on both publicly available datasets with challenging lighting conditions and fast motions, along with a new dataset with HDR reference that we provide. The proposed AKF pipeline outperforms other state-of-the-art methods in both absolute intensity error (69.4% reduction) and image similarity indexes (average 35.5% improvement). We also demonstrate the integration of image convolution with linear spatial kernels Gaussian, Sobel, and Laplacian as an application of our architecture.

Multi-Bracket High Dynamic Range Imaging with Event Cameras

“…Hybrid single-exposure HDR methods [17,56] combine an LDR image captured by a high-resolution frame camera with events acquired by an auxiliary event camera. These methods propose different ways to enhance the luminance of the LDR image using the added information from events.…”

“…To bypass these limitations, hybrid HDR works [17,56] proposed to combine a single LDR image captured by a frame camera with events from an auxiliary event camera, thus leveraging the advantages of both. The authors proposed different ways to enhance the luminance of the LDR image using the added information from events, but they still rely on the chrominance of the LDR image for color.…”

Event-guided Multi-patch Network with Self-supervision for Non-uniform Motion Deblurring

“…E-CIR [44] proposed to leverage events to construct the parametric bases, and introduced a refinement module to propagate visual features among frames. Wang et al [55] proposed to recreate intensity images using an asynchronous Kalman filter based on a unified event and frame uncertainty model. The images reconstructed using these methods, however, include artifacts due to the accumulation of event noises.…”

Pan‐risk factor for a comprehensive cardiovascular health management

“…The HDR of events makes it naturally more advantageous to reconstruct an HDR image/video. The predominant methods can be divided into two main categories: event-based HDR image/video HDR methods [89], [120], [121] and eventguided image/video HDR methods (a hybrid of event and frame data) [122], [123], [124]. Event-based image/video HDR typically employs the idea of event-to-image translation-reconstructing HDR images from events, as mentioned in Sec.…”

“…Event-guided image/video also follows these two paradigms. [122], [123], [124] explored the potential of merging both events and frames for this task, as shown in Fig. 11(c).…”