Radar Voxel Fusion for 3D Object Detection

Abstract

Automotive traffic scenes are complex due to the variety of possible scenarios, objects, and weather conditions that need to be handled. In contrast to more constrained environments, such as automated underground trains, automotive perception systems cannot be tailored to a narrow field of specific tasks but must handle an ever-changing environment with unforeseen events. As currently no single sensor is able to reliably perceive all relevant activity in the surroundings, sensor data fusion is applied to perceive as much information as possible. Data fusion of different sensors and sensor modalities on a low abstraction level enables the compensation of sensor weaknesses and misdetections among the sensors before the information-rich sensor data are compressed and thereby information is lost after a sensor-individual object detection. This paper develops a low-level sensor fusion network for 3D object detection, which fuses lidar, camera, and radar data. The fusion network is trained and evaluated on the nuScenes data set. On the test set, fusion of radar data increases the resulting AP (Average Precision) detection score by about 5.1% in comparison to the baseline lidar network. The radar sensor fusion proves especially beneficial in inclement conditions such as rain and night scenes. Fusing additional camera data contributes positively only in conjunction with the radar fusion, which shows that interdependencies of the sensors are important for the detection result. Additionally, the paper proposes a novel loss to handle the discontinuity of a simple yaw representation for object detection. Our updated loss increases the detection and orientation estimation performance for all sensor input configurations. The code for this research has been made available on GitHub.

A Comprehensive Survey of Deep Learning Multisensor Fusion-based 3D Object Detection for Autonomous Driving: Methods, Challenges, Open Issues, and Future Directions

“…This operation causes information loss. However, works like [84] directly fused camera, LiDAR, and radar inputs without initial processing.…”

“…Method Dataset Input Fusion Type mAP-BEV/NDS (%) mAP-3D/mAP (%) MV3D [4] KITTI [32] RGB image& LiDAR early, middle, late --PointFusion [85] KITTI [32] RGB image& LiDAR early -40.13 AVOD-FPN [5] KITTI [32] RGB image &LiDAR middle 64.03 55.63 SAANET [111] KITTI [32] RGB imag & LiDAR middle -52.5 3D-CVF [58] KITTI [32] RGB image &LiDAR middle (gated based) --CenterFusion [109] nuScenes [30] RGB image & radar middle 45.30 33.20 RVF-Net [84] nuScenes [30] RGB image, radar, & LiDAR early 54.86 -FusionNet [96] custom [96] RGB image & radar early -73.5 Meyer et al [82] Astyx [40] RGB image & radar late -48.0 VPFNet [124] KITTI [32] RGB If we have a deeper network, we lose much information at each layer because of this operation. Coming up with a new idea to replace a pooling operation with an equivalent operation that does not cause information loss or build a reconstruction /upsampling layer that can fully reconstruct the lost features with no loss, such as using wavelets, may help to increase the detection performance.…”

“…The output on the dataset shows that radar and camera fusion perform better than LiDAR and camera fusion. Nobis et al[84] proposed a LiDAR, camera, and radar fusion model, RadarVoxelFusionNet (RVF-Net), for 3D object detection. The LiDAR data points are projected into the image space and fused with camera images to simulate the depth camera and generate 3D points.…”

Self-Supervised Velocity Estimation for Automotive Radar Object Detection Networks

“…These point cloud based networks can be further differentiated into grid-based and point-based architectures. Grid-based approaches first render the point cloud into a 2D bird eye view (BEV) or 3D voxel grid using hand-crafted operations [11], [28], [29], [30], [31] or learned feature-encoders [32], [12], [31] and subsequently apply convolutional backbones to the grid.…”

MANet: End-to-End Learning for Point Cloud Based on Robust Pointpillar and Multiattention

“…In the field of target detection [35][36][37], recall and precision are mainly used as the performance measure of the algorithm. Precision (P) and recall (R) are, respectively, defined as follows: R = TP TP + FN ,…”