3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-view Spatial Feature Fusion for 3D Object Detection

Yoo, Jin Hyeok; Kim, Yeocheol; Kim, Ji Song; Choi, Jun Won

doi:10.1007/978-3-030-58583-9_43

Cited by 335 publications

(150 citation statements)

References 25 publications

Supporting

Mentioning

150

Contrasting

Order By: Relevance

“…Therefore, all information is retained and can potentially improve the obstacle detection accuracy. Reference [ 181 ] proposed a two-stage 3D obstacle detection architecture, named 3D-cross view fusion (3D-CVF). In the second stage, they utilized the LLF approach to fuse the joint camera-LiDAR feature map obtained from the first stage with the low-level camera and LiDAR features using a 3D region of interest (RoI)-based pooling method.…”

Section: Sensor Calibration and Sensor Fusion For Object Detectionmentioning

confidence: 99%

“…In the second stage, they utilized the LLF approach to fuse the joint camera-LiDAR feature map obtained from the first stage with the low-level camera and LiDAR features using a 3D region of interest (RoI)-based pooling method. They evaluated the proposed method on KITTI and nuScenes datasets and reported that the object detection results outperformed the state-of-the-art 3D object detectors in the KITTI leaderboard (see reference [ 181 ] for a more comprehensive summary). In practice, the LLF approach comes with a multitude of challenges, not least in its implementation.…”

Section: Sensor Calibration and Sensor Fusion For Object Detectionmentioning

confidence: 99%

See 1 more Smart Citation

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

Yeong

Velasco-Hernandez

Barry

et al. 2021

Sensors

490

133

View full text Add to dashboard Cite

With the significant advancement of sensor and communication technology and the reliable application of obstacle detection techniques and algorithms, automated driving is becoming a pivotal technology that can revolutionize the future of transportation and mobility. Sensors are fundamental to the perception of vehicle surroundings in an automated driving system, and the use and performance of multiple integrated sensors can directly determine the safety and feasibility of automated driving vehicles. Sensor calibration is the foundation block of any autonomous system and its constituent sensors and must be performed correctly before sensor fusion and obstacle detection processes may be implemented. This paper evaluates the capabilities and the technical performance of sensors which are commonly employed in autonomous vehicles, primarily focusing on a large selection of vision cameras, LiDAR sensors, and radar sensors and the various conditions in which such sensors may operate in practice. We present an overview of the three primary categories of sensor calibration and review existing open-source calibration packages for multi-sensor calibration and their compatibility with numerous commercial sensors. We also summarize the three main approaches to sensor fusion and review current state-of-the-art multi-sensor fusion techniques and algorithms for object detection in autonomous driving applications. The current paper, therefore, provides an end-to-end review of the hardware and software methods required for sensor fusion object detection. We conclude by highlighting some of the challenges in the sensor fusion field and propose possible future research directions for automated driving systems.

show abstract

Section: Sensor Calibration and Sensor Fusion For Object Detectionmentioning

confidence: 99%

Section: Sensor Calibration and Sensor Fusion For Object Detectionmentioning

confidence: 99%

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

Yeong

Velasco-Hernandez

Barry

et al. 2021

Sensors

490

133

View full text Add to dashboard Cite

show abstract

“…Therefore, all information is retained and can potentially improve the obstacle detection accuracy. Reference [178] proposed a two-stage 3D obstacle detection architecture, named 3D-cross view fusion (3D-CVF). In the second stage, they utilized the LLF approach to fuse the joint camera-LiDAR feature map obtained from the first stage with the low-level camera and LiDAR features using a 3D region of interest (RoI)-based pooling method.…”

Section: Sensor Fusion Approachesmentioning

confidence: 99%

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

Yeong¹,

Velasco-Hernandez²,

Barry³

et al. 2021

Preprint

149

View full text Add to dashboard Cite

The market for autonomous vehicles (AV) is expected to experience significant growth over the coming decades and to revolutionize the future of transportation and mobility. The AV is a vehicle that is capable of perceiving its environment and perform driving tasks safely and efficiently with little or no human intervention and is anticipated to eventually replace conventional vehicles. Self-driving vehicles employ various sensors to sense and perceive their surroundings and, also rely on advances in 5G communication technology to achieve this objective. Sensors are fundamental to the perception of surroundings and the development of sensor technologies associated with AVs has advanced at a significant pace in recent years. Despite remarkable advancements, sensors can still fail to operate as required, due to for example, hardware defects, noise and environment conditions. Hence, it is not desirable to rely on a single sensor for any autonomous driving task. The practical approaches shown in recent research is to incorporate multiple, complementary sensors to overcome the shortcomings of individual sensors operating independently. This article reviews the technical performance and capabilities of sensors applicable to autonomous vehicles, mainly focusing on vision cameras, LiDAR and Radar sensors. The review also considers the compatibility of sensors with various software systems enabling the multi-sensor fusion approach for obstacle detection. This review article concludes by highlighting some of the challenges and possible future research directions.

show abstract

“…The LIDAR-RGB 3D fusion object detection algorithm mainly includes MV3D [ 7 ], AVOD [ 3 ], 3D-CVF [ 9 ], MMF [ 10 ], etc., which are more robust in practical applications. MV3D is the first pioneer in using the fusion of LIDAR point cloud data and RGB image information.…”

Section: Related Workmentioning

confidence: 99%

“…However, the fusion in the decision-making layer has little effect on the raw data information fusion, and the confidence scores of the proposals generated by the two modules are not related. For the deep fusion-based methods, 3D-CVF [ 9 ] and MMF [ 10 ] adopt feature extractors respectively for LIDAR and image, and fuse images and LIDAR hierarchically and semantically. Finally, the semantic fusion of multi-scale information is realized.…”

Section: Introductionmentioning

confidence: 99%

Cascaded Cross-Modality Fusion Network for 3D Object Detection

Chen

Lin

Sun

et al. 2020

Sensors

View full text Add to dashboard Cite

We focus on exploring the LIDAR-RGB fusion-based 3D object detection in this paper. This task is still challenging in two aspects: (1) the difference of data formats and sensor positions contributes to the misalignment of reasoning between the semantic features of images and the geometric features of point clouds. (2) The optimization of traditional IoU is not equal to the regression loss of bounding boxes, resulting in biased back-propagation for non-overlapping cases. In this work, we propose a cascaded cross-modality fusion network (CCFNet), which includes a cascaded multi-scale fusion module (CMF) and a novel center 3D IoU loss to resolve these two issues. Our CMF module is developed to reinforce the discriminative representation of objects by reasoning the relation of corresponding LIDAR geometric capability and RGB semantic capability of the object from two modalities. Specifically, CMF is added in a cascaded way between the RGB and LIDAR streams, which selects salient points and transmits multi-scale point cloud features to each stage of RGB streams. Moreover, our center 3D IoU loss incorporates the distance between anchor centers to avoid the oversimple optimization for non-overlapping bounding boxes. Extensive experiments on the KITTI benchmark have demonstrated that our proposed approach performs better than the compared methods.

show abstract

3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-view Spatial Feature Fusion for 3D Object Detection

Cited by 335 publications

References 25 publications

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review

Cascaded Cross-Modality Fusion Network for 3D Object Detection

Contact Info

Product

Resources

About