FusionNet: Coarse-to-Fine Extrinsic Calibration Network of LiDAR and Camera with Hierarchical Point-pixel Fusion

Abstract

The accurate and robust calibration result of sensors is considered as an important building block to the follow-up research in the autonomous driving and robotics domain. The current works involving extrinsic calibration between 3D LiDARs and monocular cameras mainly focus on target-based and target-less methods. The target-based methods are often utilized offline because of restrictions, such as additional target design and target placement limits. The current target-less methods suffer from feature indeterminacy and feature mismatching in various environments. To alleviate these limitations, we propose a novel target-less calibration approach which is based on the 2D-3D edge point extraction using the occlusion relationship in 3D space. Based on the extracted 2D-3D point pairs, we further propose an occlusion-guided pointmatching method that improves the calibration accuracy and reduces computation costs. To validate the effectiveness of our approach, we evaluate the method performance qualitatively and quantitatively on real images from the KITTI dataset. The results demonstrate that our method outperforms the existing target-less methods and achieves low error and high robustness that can contribute to the practical applications relying on highquality Camera-LiDAR calibration.

A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration

“…The method of directly processing 2D and 3D data is more in line with natural laws and human information processing processes, but most of these methods are limited by the existing network design level and adopt a hybrid learning mode. Fortunately, in the past year of development, there have been a few end-to-end networks that directly process point clouds and RGB images, such as RGKCNet [ 107 ] and FusionNet [ 115 ]. Therefore, the design of calibration networks that directly process two (or even multiple) different dimensions and properties of data will be a research trend and a challenging issue in this field.…”

“…Secondly, some traditional feature-based methods require more accurate and robust feature and semantic feature extraction networks (or underlying feature that constitutes a semantic object) to enhance the performance of algorithms. With the continuous deepening of research, end-to-end calibration networks have also emerged in local feature calibration methods, such as RGKCNet [106], DedgeNet [108], and FustionNet [115]. DedgeNet projects the point cloud into a depth map and then inputs it, with the RGB image, into an end-to-end network.…”

“…Then, the parameters matrix between the LiDAR and the camera is predicted. In 2023, Wang et al proposed a similar calibration framework in FusionNet [ 115 ], but extracted key points from 3D point clouds instead of images. The 3D points were projected onto RGB images, and fused.…”

“…3D point feature extraction and fusion. In 2023, G. Wang et al proposed a similar calibration framework in FusionNet [ 115 ], in which point features at different resolutions are directly extracted from the point cloud through PointNet++, and key points are projected onto the camera imaging plane to obtain the corresponding points in the RGB image and their descriptions in the global feature map, and fused. The parameters are predicted in a coarse to fine estimation framework, as shown in Figure 21 .…”

“…With the continuous deepening of research, end-to-end calibration networks have also emerged in local feature calibration methods, such as RGKCNet [ 106 ], DedgeNet [ 108 ], and FustionNet [ 115 ]. DedgeNet projects the point cloud into a depth map and then inputs it, with the RGB image, into an end-to-end network.…”