Low-Level Sensor Fusion for 3D Vehicle Detection Using Radar Range-Azimuth Heatmap and Monocular Image

Kim, Jinhyeong; Kim, Young-Seok; Kum, Dongsuk

doi:10.1007/978-3-030-69535-4_24

Cited by 13 publications

(20 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, feature map fusion utilises two encoders to map radar and images into the same latent space with high-level semantics. The detection frameworks are flexible, including one-stage methods [179,180] and two-stage methods [33,181,182]. The one-stage methods leverage two branches of neural networks to extract feature maps from radar and images, respectively, and then concatenate the feature maps together.…”

Section: Feature Map Fusionmentioning

confidence: 99%

Towards Deep Radar Perception for Autonomous Driving: Datasets, Methods, and Challenges

Zhou

Liu

Zhao

et al. 2022

Sensors

View full text Add to dashboard Cite

With recent developments, the performance of automotive radar has improved significantly. The next generation of 4D radar can achieve imaging capability in the form of high-resolution point clouds. In this context, we believe that the era of deep learning for radar perception has arrived. However, studies on radar deep learning are spread across different tasks, and a holistic overview is lacking. This review paper attempts to provide a big picture of the deep radar perception stack, including signal processing, datasets, labelling, data augmentation, and downstream tasks such as depth and velocity estimation, object detection, and sensor fusion. For these tasks, we focus on explaining how the network structure is adapted to radar domain knowledge. In particular, we summarise three overlooked challenges in deep radar perception, including multi-path effects, uncertainty problems, and adverse weather effects, and present some attempts to solve them.

show abstract

Section: Feature Map Fusionmentioning

confidence: 99%

Towards Deep Radar Perception for Autonomous Driving: Datasets, Methods, and Challenges

Zhou

Liu

Zhao

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…For 3D object detection, the authors of [18] propose GRIF-Net to fuse radar and camera data. After individual processing, the feature fusion is performed by a gated region of interest fusion (GRIF).…”

Section: Camera Radar Fusionmentioning

confidence: 99%

Radar Voxel Fusion for 3D Object Detection

et al. 2021

View full text Add to dashboard Cite

Automotive traffic scenes are complex due to the variety of possible scenarios, objects, and weather conditions that need to be handled. In contrast to more constrained environments, such as automated underground trains, automotive perception systems cannot be tailored to a narrow field of specific tasks but must handle an ever-changing environment with unforeseen events. As currently no single sensor is able to reliably perceive all relevant activity in the surroundings, sensor data fusion is applied to perceive as much information as possible. Data fusion of different sensors and sensor modalities on a low abstraction level enables the compensation of sensor weaknesses and misdetections among the sensors before the information-rich sensor data are compressed and thereby information is lost after a sensor-individual object detection. This paper develops a low-level sensor fusion network for 3D object detection, which fuses lidar, camera, and radar data. The fusion network is trained and evaluated on the nuScenes data set. On the test set, fusion of radar data increases the resulting AP (Average Precision) detection score by about 5.1% in comparison to the baseline lidar network. The radar sensor fusion proves especially beneficial in inclement conditions such as rain and night scenes. Fusing additional camera data contributes positively only in conjunction with the radar fusion, which shows that interdependencies of the sensors are important for the detection result. Additionally, the paper proposes a novel loss to handle the discontinuity of a simple yaw representation for object detection. Our updated loss increases the detection and orientation estimation performance for all sensor input configurations. The code for this research has been made available on GitHub.

show abstract

“…The primary reason behind autonomous cars not being more commonplace is their dependence on lidars, cameras, or their fusion, which are unable to perform robustly in cases of occlusions and adverse weather conditions [2,11]. This shortcoming, in cameras and lidar, has sparked a major interest in automotive radar-based sensing, particularly in camera radar fusion systems [17,26,28].…”

Section: Introductionmentioning

confidence: 99%

“…More advanced and state-of-the-art approaches perform fusion at feature level. For example AVOD-fusion [17] first simultaneously extracts features from camera perspective and radar bird's eye view (BEV) and then fuses them on per object basis, to take advantage of both the sensors in their views. However, we find that the simultaneous feature extraction and fusion approach does not account for the cases where camera become unreliable.…”

Section: Introductionmentioning

confidence: 99%

RadSegNet: A Reliable Approach to Radar Camera Fusion

Rungta¹,

Bharadia²

2022

Preprint

View full text Add to dashboard Cite

Perception systems for autonomous driving have seen significant advancements in their performance over last few years. However, these systems struggle to show robustness in extreme weather conditions because sensors like lidars and cameras, which are the primary sensors in a sensor suite, see a decline in performance under these conditions. In order to solve this problem, cameraradar fusion systems provide a unique opportunity for all weather reliable high quality perception. Cameras provides rich semantic information while radars can work through occlusions and in all weather conditions. In this work we show that the state-of-the-art fusion methods perform poorly when camera input is degraded, which essentially results in losing the all-weather reliability they set out to achieve. Contrary to these approaches, we propose a new method, RadSeg-Net, that uses a new design philosophy of independent information extraction and truly achieves reliability in all conditions including occlusions and adverse weather. We develop and validate our proposed system on the benchmark Astyx dataset and further verify these results on the RADIATE dataset. When compared to state-of-the-art methods, RadSegNet achieves a 27% improvement on Astyx and 41.46% increase on RADIATE, in average precision score and maintains a significantly better performance in adverse weather conditions.

show abstract

Low-Level Sensor Fusion for 3D Vehicle Detection Using Radar Range-Azimuth Heatmap and Monocular Image

Cited by 13 publications

References 20 publications

Towards Deep Radar Perception for Autonomous Driving: Datasets, Methods, and Challenges

Towards Deep Radar Perception for Autonomous Driving: Datasets, Methods, and Challenges

Radar Voxel Fusion for 3D Object Detection

RadSegNet: A Reliable Approach to Radar Camera Fusion

Contact Info

Product

Resources

About