Real-Time Optical Flow for Vehicular Perception with Low- and High-Resolution Event Cameras

Abstract

Event cameras capture changes of illumination in the observed scene rather than accumulating light to create images. Thus, they allow for applications under high-speed motion and complex lighting conditions, where traditional framebased sensors show their limits with blur and over-or underexposed pixels. Thanks to these unique properties, they represent nowadays an highly attractive sensor for ITS-related applications. Event-based optical flow (EBOF) has been studied following the rise in popularity of these neuromorphic cameras. The recent arrival of high-definition neuromorphic sensors, however, challenges the existing approaches, because of the increased resolution of the events pixel array and a much higher throughput. As an answer to these points, we propose an optimized framework for computing optical flow in real-time with both low-and high-resolution event cameras. We formulate a novel dense representation for the sparse events flow, in the form of the "inverse exponential distance surface". It serves as an interim frame, designed for the use of proven, state-of-the-art frame-based optical flow computation methods. We evaluate our approach on both low-and high-resolution driving sequences, and show that it often achieves better results than the current state of the art, while also reaching higher frame rates, 250Hz at 346×260 pixels and 77Hz at 1280×720 pixels.Index Terms-Machine vision, neuromorphic cameras, optical flow, real-time applications.• a specific pipeline-based architecture, for computing realtime optical flow using the events from low-or highresolution neuromorphic sensors; • the formulation of a novel dense "inverse exponential distance surface", that acts as the frame-based representation computed from the events, able to feed any image-based optical flow method; • a coherent choice of algorithms and methods together for all the steps up to the fast frame-based state-of-theart optical flow (with temporal smoothing to fit well with potentially noisy input events); • we finally build and share a complementary highdefinition event-based dataset of indoor sequences with high-speed movements, used as part of our evaluation.Videos accompanying this article, showing results for both low-and high-resolution data, are available at https://youtube.com/playlist?list= PLLL0eWAd6OXBRXli-tB1NREdhBElAxisD.1 Considering a car at 120Km/h, if we tolerate to drive a distance of 1 meter to achieve perception analysis, it means a maximum latency of 30ms.