Live demonstration: Behavioural emulation of event-based vision sensors

Abstract

This demonstration shows how an inexpensive high frame-rate USB camera is used to emulate existing and proposed activity-driven event-based vision sensors. A PS3-Eye camera which runs at a maximum of 125 frames/second with colour QVGA (320×240) resolution is used to emulate several event-based vision sensors, including a Dynamic Vision Sensor (DVS), a colour-change sensitive DVS (cDVS), and a hybrid vision sensor with DVS+cDVS pixels. The emulator is integrated into the jAER software project for event-based real-time vision and is used to study use cases for future vision sensor designs. I. DEMONSTRATIONRecent developments in activity-driven, event-based vision sensors have opened up a promising alternative to conventional frame-based vision. By outputting a sparse and variable-rate stream of data originating asynchronously from pixels with local gain control, these sensors reduce processing cost and latency, while increasing dynamic range. Except for [1], these vision sensor developments have been preceded by silicon chip developments. We thought that it would be useful to develop a behavioural emulator of such chip architectures that would allow us or others to study how to use a proposed vision sensor in application scenarios, and to allow us to model, in software, non-idealities such as fixed pattern noise and temporal noise. In particular, we needed an emulator that is fully integrated into our host-side software infrastructure [3]. In this demonstration, we show how this emulation allows us to model several existing and proposed vision sensor architectures. II. DEMONSTRATION SETUPThe demonstration setup consists of a laptop, a webcam (Playstation PS3-Eye), and a DVS128 Dynamic Vision Sensor [4]. A large monitor or beamer shows the output. III. VISITOR EXPERIENCEVisitors see how the emulator models an existing DVS128, how the improved resolution of the QVGA emulation will benefit applications in wide-area surveillance, and how the emulation models the cDVS sensor we are developing to simultaneously detect colour change and brightness change [5] . By using a high speed rotating stimulus, visitors also see the limitations of the conventional camera in terms of time resolution (8ms for the PS3-Eye compared with 1us for the DVS128), and in terms of dynamic range (<50dB for the PS3-Eye, compared with 120dB for the DVS128). IV. DEMONSTRATION READINESSThe demonstration is fully functional (see Fig. 1). We can simultaneously run the emulation and a real event-based sensor, record data, and play it back. We can display data in a variety of formats such as the spatial histogram shown in Vision sensor emulation demonstration. The output of jAER simultaneously shows the emulated DVS+cDVS and the raw PS3-Eye camera. Left window shows output of jAER [3]. The emulated pixels respond both to brightness change (black and white pixels) and mean wavelength change (red and blue pixels). The right window shows the control panel which allows camera and emulation parameter control, and also shows the raw PS3-Eye came...

Event-Based Noise Filtration with Point-of-Interest Detection and Tracking for Space Situational Awareness

“…The simulator generates a number of spikes equal to the intensity difference divided by the chosen threshold, which are then assigned timestamps either all synchronous with the current image frame or linearly interpolated over the time period between frames. This linear interpolation would result in the generated spikes being equally distributed over the time period between frames [27]. This approach was later applied towards generating artificial, event-based datasets for visual odometry and SLAM algorithms hoping to leverage event-based techniques.…”

“…Luminance values at each pixel are first calculated using the 'perceptual' luminance of the pixel and its immediate neighborhood, followed by taking a weighted average of values in surrounding pixels. The spike trains are then generated according to similar manner as in [27], but with the threshold for event generation being compared to the minimum of a neighborhood of values rather than a single pixel. Upon validation testing, using LICE pixel values resulted in the simulated event stream having events temporally and spatially distributed more similarly to a true event stream, albeit with worse performance than log-intensity approach [29].…”

Tracking from a moving platform with the Dynamic Vision Sensor

“…Afterwards, the confusion matrix elements were averaged across all frames, which gave mean values of 10,612, 23,904, 4,890, and 3,794 true positive, true negative, false positive, and false negative event counts, respectively. Similar metrics could be computed with different DVS models 2,7 in this scenario to compare their fidelity.…”

pyDVS: An extensible, real-time Dynamic Vision Sensor emulator using off-the-shelf hardware

“…Since we are developing a real-time system, all events should be emitted between frames, so the maximum number of spikes per neuron per frame is N b . As in previous emulators [5], our basic emulator's output format is rate-based. Each emitted spike signifies the pixel changed by H brightness levels, where H is also the threshold.…”

“…An alternative that could reduce the cost and scarcity of DVSs while keeping spike rates low is to emulate the behaviour of a DVS. Katz et al developed a DVS emulator to test behaviours for new sensor models [5]. In their work, they transform video [at 125 frames per second (FPS)] provided by a commercial camera into a spike stream.…”