RADIANT: Radar-Image Association Network for 3D Object Detection

Abstract

As a direct depth sensor, radar holds promise as a tool to improve monocular 3D object detection, which suffers from depth errors, due in part to the depth-scale ambiguity. On the other hand, leveraging radar depths is hampered by difficulties in precisely associating radar returns with 3D estimates from monocular methods, effectively erasing its benefits. This paper proposes a fusion network that addresses this radar-camera association challenge. We train our network to predict the 3D offsets between radar returns and object centers, enabling radar depths to enhance the accuracy of 3D monocular detection. By using parallel radar and camera backbones, our network fuses information at both the feature level and detection level, while at the same time leveraging a state-of-the-art monocular detection technique without retraining it. Experimental results show significant improvement in mean average precision and translation error on the nuScenes dataset over monocular counterparts. Our source code is available at https://github.com/longyunf/radiant.

ConCs-Fusion: A Context Clustering-Based Radar and Camera Fusion for Three-Dimensional Object Detection

“…Camera can provide appearance information of targets, whereas radar can provide parameters such as distance and velocity of targets, and is less affected by extreme weather conditions. RADIANT [2] is a 3D object detection method based on the FCOS3D framework, which addresses the problem of radarcamera association. It can predict the 3D offset between radar data and the center point of the real object.…”

“…Moreover, 2D object detection does not involve orientation-related parameters [1]. Although 2D object detection is well-established, the literature [2,3] has demonstrated that methods achieving excellent results in 2D object detection tasks can experience a drastic drop in performance in 3D object detection. The realworld scenarios for autonomous driving technology involve a 3D environment; thus, 3D object detection is a topic worth exploring in depth.…”

“…We propose a ConCs-Fusion network that leverages a fusion module to aggregate and redistribute features, enabling mutual assistance between the radar and camera. (2) In the fusion module, we employ transposed convolution to learn different scale features from context cluster blocks, and fuse the output features through concatenation. We then utilize the nonlinear representation capability of multi-layer perceptron (MLP) to reduce the dimensionality of fused features, thereby improving the computational efficiency of the model.…”

Radars for Autonomous Driving: A Review of Deep Learning Methods and Challenges

“…The camera-radar early fusion models in the literature primarily use either a perspective view [89], [91] or a BEV [92], [93], depending on their application. Some models, however, use non-standard joint views that offer a balanced compromise in view disparity for both sensors, such as Cram-Net [28].…”

“…RADIANT [91] addresses the issue of imprecise association of radar points with camera object detections, which can lead to sub-optimal depth estimations. Instead of associating radar points to objects as other models do, RADIANT learns to predict 3D offsets between radar points and object centers, followed by a feedforward depth weighing network to refine their monocular estimated depth.…”