CalibBD: Extrinsic Calibration of the LiDAR and Camera Using a Bidirectional Neural Network

Abstract

With the rapid growth of self-driving vehicles, automobiles demand diverse data from multiple sensors to perceive the surrounding environment. Calibrating preprocessing between multiple sensors is necessary to utilize the data effectively. In particular, the LiDAR-camera pair, a suitable complement with 2D-3D information for each other, has been widely used in autonomous vehicles. Most traditional calibration methods require specific calibration targets set up under complicated environmental conditions, which require expensive human manual work. In this study, we propose a deep neural network that does not require any specific targets and offline setup to find the six degrees of freedom (6 DoF) transformation between LiDAR and the camera. Unlike previous deep learning CNN-based methods, which use raw 3D point clouds and 2D images frame by frame, CalibBD utilizes Bi-LSTM for sequence data to extract temporal features between consecutive frames. It not only predicts the calibration parameters by minimizing both transformation and depth losses but also calibrates the camera parameters by using temporal loss to refine the calibration parameters. The proposed model achieves a steady performance under various deviations of mis-calibration parameters and achieved higher results in terms of accuracy than the state-of-the-art CNN-based method on the KITTI datasets.

A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration

“…Considering that the calibration, especially online calibration, is a time-varying process with correlations between data frames, Shi et al proposed the CalibRCNN [ 57 ] in 2020, applying a recurrent neural network, long short term memory network (LSTM), to extract temporal characteristics, and estimating extrinsic parameters through the constraint of the time. In 2022, Nguyen et al replaced LSTM in CalibRCNN with a Bi-LSTM module and proposed CalibBD [ 63 ]. Similarly, Zhu et al applied LSTM to point cloud depth maps and RGB depth maps in CalibDepth [ 47 ], thereby improving the accuracy and robustness of the algorithm.…”

“…In this type of method, it is considered that extrinsic parameters estimation is not only a spatial but also a temporal estimation problem, as shown in Figure 13 a. Based on this, CalibRCNN [ 57 ] and some other methods [ 47 , 63 ] all use LSTM structure for matching, as shown in Figure 13 b. During the matching process, the matching results at the previous time affect the current matching.…”

“…This loss is defined as the dense pixel-wise error (since each pixel is encoded with the depth intensity) between the predicted and correct depth maps, as that in CalibNet [ 44 ]. CalibDepth [ 47 ] and CalibBD [ 63 ] also provide similar definitions, but with different names. In addition, CFNet [ 59 ] and DXQ-Net [ 41 ] define it as calibration flow.…”

“…Epipolar geometry constraint loss. Similarly, CalibRCNN [ 57 ] and CalibBD [ 63 ] also take the influence of time series into account. Changes in camera pose can be obtained from changes in point cloud with predicted extrinsic parameters.…”

“…Ignoring the mis-calibrated range factor, we find that Z. Zhang's method [70] demonstrated a mean TAE of 0.33 cm and PseudoCal [69] gave a mean RAE of 0.05 • . Ded-geNet [108] and CalibDB [63] also win the single TAE or RAE competition, respectively. From the results data, it can be seen that LCCNet [46], SCNet [94], and ATOP [40] also had good performance.…”