Real-Time Efficient FPGA Implementation of the Multi-Scale Lucas-Kanade and Horn-Schunck Optical Flow Algorithms for a 4K Video Stream

Abstract

The information about optical flow, i.e., the movement of pixels between two consecutive images from a video sequence, is used in many vision systems, both classical and those based on deep neural networks. In some robotic applications, e.g., in autonomous vehicles, it is necessary to calculate the flow in real time. This represents a challenging task, especially for high-resolution video streams. In this work, two gradient-based algorithms—Lucas–Kanade and Horn–Schunck—were implemented on a ZCU 104 platform with Xilinx Zynq UltraScale+ MPSoC FPGA. A vector data format was used to enable flow calculation for a 4K (Ultra HD, 3840 × 2160 pixels) video stream at 60 fps. In order to detect larger pixel displacements, a multi-scale approach was used in both algorithms. Depending on the scale, the calculations were performed for different data formats, allowing for more efficient processing by reducing resource utilisation. The presented solution allows real-time optical flow determination in multiple scales for a 4K resolution with estimated energy consumption below 6 W. The algorithms realised in this work can be a component of a larger vision system in advanced surveillance systems or autonomous vehicles.

A Self-Organizing Multi-Layer Agent Computing System for Behavioral Clustering Recognition

“…Most hardware implementations of optical flow have focused on the HS algorithm and the LK algorithm [25]. For example, Lazcano et al [26] processed 320 × 240 pixel video frames at 4 FPS by modifying the classical HSOF method on a GPU GeForce NVIDIA-GTX-980-Ti platform.…”

A Self-Organizing Multi-Layer Agent Computing System for Behavioral Clustering Recognition

“…Most hardware implementations of optical flow have focused on the HS algorithm and the LK algorithm [ 25 ]. For example, Lazcano et al [ 26 ] processed pixel video frames at 4 FPS by modifying the classical HSOF method on a GPU GeForce NVIDIA-GTX-980-Ti platform.…”

A Fast and Accurate Optical Flow Camera for Resource-Constrained Edge Applications

“…PX4FLOW a near-sensor accelerator, which consists of a camera and an ARM Cortex M4F micro-controller for optical flow prediction has been published almost a decade ago [20]. Recent works show FPGA implementations for the prediction of optical flow applying both traditional computer vision algorithms [2] as well as machine learning algorithms [21]. Furthermore, [22] presents a hardware accelerator for the prediction of optical flow from event camera data.…”

“…It tracks the movement of single features in an image sequence or stream. Optical flow can also be applied in a dense manner to track the movement of every pixel in a frame [2].…”