Implementation of the PointPillars Network for 3D Object Detection in Reprogrammable Heterogeneous Devices Using FINN

Abstract

In this paper, we present a hardware-software implementation of a deep neural network for object detection based on a point cloud obtained by a LiDAR sensor. The PointPillars network was used in the research, as it is a reasonable compromise between detection accuracy and calculation complexity. The Brevitas / PyTorch tools were used for network quantisation (described in our previous paper) and the FINN tool for hardware implementation in the reprogrammable Zynq UltraScale+ MPSoC device. The obtained results show that quite a significant computation precision limitation along with a few network architecture simplifications allows the solution to be implemented on a heterogeneous embedded platform with maximum 19% AP loss in 3D, maximum 8% AP loss in BEV and execution time 375ms (the FPGA part takes 262ms). We have also compared our solution in terms of inference speed with a Vitis AI implementation proposed by Xilinx (19 Hz frame rate). Especially, we have thoroughly investigated the fundamental causes of differences in the frame rate of both solutions. The code is available at https://github.com/vision-agh/pp-finn.

PointPillars Backbone Type Selection For Fast and Accurate LiDAR Object Detection

“…In the results, the original PointPillars network is denoted as "base". We adopt the real-time definition from [27], i.e. processing point clouds at a rate of 10 fps or greater.…”

“…In the case of PointPillars, this increases the potential to implement the algorithm in a real-time embedded system while maintaining reasonable high detection performance. Our previous work [27] shows that such an implementation for the original version of the algorithm is very difficult if not impossible.…”

“…by using it as part of CenterPoint and implement it on an embedded system, e.g. on an SoC FPGA as a continuation of our previous work [27].…”

“…Based on the initial analysis, we have selected the PointPillars [15] network for experiments, mainly due to the favourable ratio of detection precision to computational complexity. In our previous work [27] we have described the process of running the PointPillars network on an embedded platform -the ZCU104 evaluation board. However, we were not able to obtain real-time performance -a single-point cloud was processed in 374.66 ms, while it should be in less than 100ms.…”

A Novel Method for 3D Object Detection in Open-Pit Mine Based on Hybrid Solid-State LiDAR Point Cloud

“…However, the projection process inevitably leads to the loss of certain geometric spatial information, resulting in shortcomings in depth prediction. In this context, PointPillars [10,11] has garnered widespread attention due to its ability to strike a favorable balance between inference speed and detection accuracy. PointPillars achieves efficient object detection by converting point cloud data into a compact voxel representation and employing a columnar structure-based processing approach.…”