Direct Image to Point Cloud Descriptors Matching for 6-DOF Camera Localization in Dense 3D Point Clouds

Abstract

We propose a novel concept to directly match feature descriptors extracted from RGB images, with feature descriptors extracted from 3D point clouds. We use this concept to localize the position and orientation (pose) of the camera of a query image in dense point clouds. We generate a dataset of matching 2D and 3D descriptors, and use it to train a proposed Descriptor-Matcher algorithm. To localize a query image in a point cloud, we extract 2D keypoints and descriptors from the query image. Then the Descriptor-Matcher is used to find the corresponding pairs 2D and 3D keypoints by matching the 2D descriptors with the preextracted 3D descriptors of the point cloud. This information is used in a robust pose estimation algorithm to localize the query image in the 3D point cloud. Experiments demonstrate that directly matching 2D and 3D descriptors is not only a viable idea but also achieves competitive accuracy compared to other state-of-the-art approaches for camera pose localization.

Unconstrained Matching of 2D and 3D Descriptors for 6-DOF Pose Estimation

“…Our preliminary work [Nadeem et al, 2019] was the first technique to estimate the 6-DOF pose for query cameras by directly matching features extracted from 2D images and 3D point clouds. Feng et al [2019] trained a deep convolutional network with triplet loss to estimate descriptors for patches extracted from images and point clouds.…”

“…A preliminary version of this work appeared in Nadeem et al [2019]. To the best of our knowledge, Nadeem et al [2019] was the first work: (i) to match directly 3D descriptors extracted from dense point clouds with the 2D descriptors from RGB images, and (ii) to use direct matching of 2D and 3D descriptors to localize camera pose with 6-DOF in dense 3D point clouds. This work extends Nadeem et al [2019] by improving all the elements of the proposed technique, including dataset generation, Descriptor-Matcher and pose estimation.…”

“…To the best of our knowledge, Nadeem et al [2019] was the first work: (i) to match directly 3D descriptors extracted from dense point clouds with the 2D descriptors from RGB images, and (ii) to use direct matching of 2D and 3D descriptors to localize camera pose with 6-DOF in dense 3D point clouds. This work extends Nadeem et al [2019] by improving all the elements of the proposed technique, including dataset generation, Descriptor-Matcher and pose estimation. Additionally, we propose a reliable method to create a large collection of 2D and 3D points with matching locations in images and point cloud, respectively.…”

“…Additionally, we propose a reliable method to create a large collection of 2D and 3D points with matching locations in images and point cloud, respectively. We also present more details of the proposed method and also evaluate the proposed technique on more challenging datasets compared to Nadeem et al [2019].…”

Direct 3D model-based tracking in omnidirectional images robust to large inter-frame motion

“…Since it is hard to find more recent works of omnidirectional image direct alignment with a 3D model of an environment, [14], [15] are considered as baselines. Despite the variety of contributions in the field of neural networks, only conventional or rectified images are considered as input of pose detection [16] approaches. One could generate conventional images from omnidirectional ones [17] to feed the latter methods but, in this paper, we focus on using acquired images directly without geometric pre-transformation.…”