Digital coded exposure formation of frames from event-based imagery

Abstract

Event-driven neuromorphic imagers have a number of attractive properties including low-power consumption, high dynamic range, the ability to detect fast events, low memory consumption and low band-width requirements. One of the biggest challenges with using event-driven imagery is that the field of event data processing is still embryonic. In contrast, decades worth of effort have been invested in the analysis of frame-based imagery. Hybrid approaches for applying established frame-based analysis techniques to event-driven imagery have been studied since event-driven imagers came into existence. However, the process for forming frames from event-driven imagery has not been studied in detail. This work presents a principled digital coded exposure approach for forming frames from event-driven imagery that is inspired by the physics exploited in a conventional camera featuring a shutter. The technique described in this work provides a fundamental tool for understanding the temporal information content that contributes to the formation of a frame from event-driven imagery data. Event-driven imagery allows for the application of arbitrary virtual digital shutter functions to form the final frame on a pixel-by-pixel basis. The proposed approach allows for the careful control of the spatio-temporal information that is captured in the frame. Furthermore, unlike a conventional physical camera, event-driven imagery can be formed into any variety of possible frames in post-processing after the data is captured. Furthermore, unlike a conventional physical camera, coded-exposure virtual shutter functions can assume arbitrary values including positive, negative, real, and complex values. The coded exposure approach also enables the ability to perform applications of industrial interest such as digital stroboscopy without any additional hardware. The ability to form frames from event-driven imagery in a principled manner opens up new possibilities in the ability to use conventional frame-based image processing techniques on event-driven imagery.

An event-driven light field, digital coded exposure imager architecture for per-pixel image regeneration in post processing

“…The concept of the digital coded exposure for event-based imagers was originally introduced in [12]. Digital coded exposure transforms a time-series of events into a single scalar value.…”

“…The value of the (x, y)th pixel of the coded exposure image for a matrix of event-streams captured from the silicon retina was calculated using the equation below for various frequencies and phases. Additional technical details on the digital coded exposure and different ways the digital coded exposure can be configured/used, can be found in [12]. Event-based re-focusing and re-exposure is achieved in post-processing by applying digital refocusing techniques to images made using digital coded exposure.…”

“…The flutter shutter expanded the scope of what shutters could do in photography, however, its hardware nature prevented post processing of the resulting images once the data was collected. In 2022 the concept of the digital coded exposure for event based imaging was introduced [12]. The digital coded exposure for event-based imaging expanded the work of the flutter shutter in a couple of important ways.…”