A Residual Network and FPGA Based Real-Time Depth Map Enhancement System

Abstract

Depth maps obtained through sensors are often unsatisfactory because of their low-resolution and noise interference. In this paper, we propose a real-time depth map enhancement system based on a residual network which uses dual channels to process depth maps and intensity maps respectively and cancels the preprocessing process, and the algorithm proposed can achieve real-time processing speed at more than 30 fps. Furthermore, the FPGA design and implementation for depth sensing is also introduced. In this FPGA design, intensity image and depth image are captured by the dual-camera synchronous acquisition system as the input of neural network. Experiments on various depth map restoration shows our algorithms has better performance than existing LRMC, DE-CNN and DDTF algorithms on standard datasets and has a better depth map super-resolution, and our FPGA completed the test of the system to ensure that the data throughput of the USB 3.0 interface of the acquisition system is stable at 226 Mbps, and support dual-camera to work at full speed, that is, 54 fps@ (1280 × 960 + 328 × 248 × 3).

FPGA Implementation of CNN Based DepthEstimation Network: MiDaSNet

“…Although it results in fast and accurate depth estimation with relatively lower computational power consumption, the depth map calculated from MLAs is sparse and an increase in pixel density, would result in excessive resource consumption and higher delays. [11] proposes a lightweight residual convolutional neural network implemented on FPGA for depth enhancement. As the name suggests, the method requires prior coarse depth information, which limits the use case of the study.…”