hARMS: A Hardware Acceleration Architecture for Real-Time Event-Based Optical Flow

Abstract

Event-based vision sensors produce asynchronous event streams with high temporal resolution based on changes in the visual scene. The properties of these sensors allow for accurate and fast calculation of optical flow as events are generated. Existing solutions for calculating optical flow from event data either fail to capture the true direction of motion due to the aperture problem, do not use the high temporal resolution of the sensor, or are too computationally expensive to be run in real time on embedded platforms. In this research, we first present a faster version of our previous algorithm, ARMS (Aperture Robust Multi-Scale flow). The new optimized software version (fARMS) significantly improves throughput on a traditional CPU. Further, we present hARMS, a hardware realization of the fARMS algorithm allowing for real-time computation of true flow on low-power, embedded platforms. The proposed hARMS architecture targets hybrid system-on-chip devices and was designed to maximize configurability and throughput. The hardware architecture and fARMS algorithm were developed with asynchronous neuromorphic processing in mind, abandoning the common use of an event frame and instead operating using only a small history of relevant events, allowing latency to scale independently of the sensor resolution. This change in processing paradigm improved the estimation of flow directions by up to 73% compared to the existing method and yielded a demonstrated hARMS throughput of up to 1.21 Mevent/s on the benchmark configuration selected. This throughput enables real-time performance and makes it the fastest known realization of aperture-robust, event-based optical flow to date.INDEX TERMS Event-based, aperture robust, optical flow, neuromorphic computing, field programmable gate arrays, system-on-chip, parallel acceleration, real-time systems

EDFLOW: Event Driven Optical Flow Camera With Keypoint Detection and Adaptive Block Matching

“…The PRM requires 5 times the sensor output bandwidth, and estimating the gradient from only 4 neighbours is sensitive to noise. Most recently, Stumpp et al [45] posted hardware Aperture Robust Multiscale flow (hARMS), an FPGA realization of [28]. This LP-based method selects the maximum flow vector from multiple scales to mitigate the aperture problem.…”

“…The Huang et al [47] flow camera relies on intensity samples, and Haessig et al [48] is computed mostly in software. The multiaperture LP code used in hARMS [45] is not published in [45] or [28]. We used exactly the same algorithm parameters for all sequences.…”

hARMS: A Hardware Acceleration Architecture for Real-Time Event-Based Optical Flow

“…In previous research, we presented the faster ARMS (fARMS) software-optimized version of ARMS as well as the hardware ARMS (hARMS) architecture for high-speed eventbased optical flow on an FPGA. The hARMS architecture is able to achieve up to 25% reduction in average endpoint error while computing the event-based optical flow asynchronously at up to 1.21 Mevent/s [23]. There have also been machinelearning approaches to generating event-based true flow.…”

“…1) Transmitter: The transmitter-subsystem complexity is almost entirely comprised of the computation of optical flow. It was demonstrated in [23] that real-time performance can be achieved for event-based optical flow on embedded platforms with the FPGA accelerated architecture. EDFlow introduced in [29] also enables real-time computation of event-based flow on FPGA, however, it does not operate purely asynchronously.…”

“…EDFlow introduced in [29] also enables real-time computation of event-based flow on FPGA, however, it does not operate purely asynchronously. Beyond the complexity of these optical-flow calculations, which is explored in detail in [23] and [29], the FBC algorithm introduces very little additional complexity on the transmitter side. The only additional workload is the operation of the system timing state machine and the event selector to determine what events are sent.…”