CMRNet: Camera to LiDAR-Map Registration

Abstract

In this paper we present CMRNet, a realtime approach based on a Convolutional Neural Network (CNN) to localize an RGB image of a scene in a map built from LiDAR data. Our network is not trained in the working area, i.e., CMRNet does not learn the map. Instead it learns to match an image to the map. We validate our approach on the KITTI dataset, processing each frame independently without any tracking procedure. CMRNet achieves 0.27m and 1.07 • median localization accuracy on the sequence 00 of the odometry dataset, starting from a rough pose estimate displaced up to 3.5m and 17 • . To the best of our knowledge this is the first CNN-based approach that learns to match images from a monocular camera to a given, preexisting 3D LiDAR-map.

Convolutional neural network-based coarse initial position estimation of a monocular camera in large-scale 3D light detection and ranging maps

“…It was focusing on localization of a camera in a local place like hospital and church, and was not suitable for city scale pose regression. Another work of tracking 6-DoF camera pose, CMRNet 24 , need an initial estimate from a global navigation satellite system device. For precisely tracking, it compared RGB images and a synthesized depth image projected from a LiDAR map.…”

Data‐driven protection levels for camera and 3D map‐based safe urban localization

“…Recently, data‐driven techniques based on deep neural networks (DNN) have demonstrated state‐of‐the‐art performance in determining the state of the camera sensor, comprised of its position and orientation, by identifying and matching patterns in images with a known map of the environment (Cattaneo et al., 2019; Lyrio et al., 2015; Oliveira et al., 2020) or an existing database of images (Sarlin et al., 2019; Taira et al., 2021).…”

“…Previously, CMRNet (Cattaneo et al., 2019) has been proposed as a DNN‐based approach for determining the vehicular state from camera images and a LiDAR‐based 3D map. In our approach, we extend the DNN architecture proposed in Cattaneo et al.…”

“…In our approach, we extend the DNN architecture proposed in Cattaneo et al. (2019) to model the position error and the covariance matrix (aleatoric uncertainty) in the vehicular state estimate.…”

“…Our main contributions are as follows: 1.We extend the CMRNet (Cattaneo et al., 2019) architecture to model both the position error in the vehicular state estimate and the associated covariance matrix. Using the 3D LiDAR‐based map of the environment, we first construct a local map representation with respect to the vehicular state estimate.…”