Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC

Abstract

We describe a learning-based system that estimates the camera position and orientation from a single input image relative to a known environment. The system is flexible w.r.t. the amount of information available at test and at training time, catering to different applications. Input images can be RGB-D or RGB, and a 3D model of the environment can be utilized for training but is not necessary. In the minimal case, our system requires only RGB images and ground truth poses at training time, and it requires only a single RGB image at test time. The framework consists of a deep neural network and fully differentiable pose optimization. The neural network predicts so called scene coordinates, i.e. dense correspondences between the input image and 3D scene space of the environment. The pose optimization implements robust fitting of pose parameters using differentiable RANSAC (DSAC) to facilitate end-to-end training. The system, an extension of DSAC++ and referred to as DSAC*, achieves state-of-the-art accuracy on various public datasets for RGB-based re-localization, and competitive accuracy for RGB-D based re-localization.

Using Image Sequences for Long-Term Visual Localization

“…Another recent line of work on single-image localization has focused on machine learning [7,8,10,11,14,15,37,38,57,75,86,88]. Scene coordinate regression approaches [7,8,14,15,57,75,88] train a random forest or convolutional neural network (CNN) to predict the corresponding 3D coordinate for each pixel.…”

Rigidity Preserving Image Transformations and Equivariance in Perspective

“…For neural networks that estimate 3D properties from images, e.g., 6D object pose estimation [10,29,41] and visual localization [8,38,46], data augmentation is a delicate matter since while the input image is transformed, the 3D target also has to undergo a corresponding transformation. As an example, in 6D object pose estimation, an image rotation around the principal point corresponds to a 3D rotation around the optical axis of the object pose, cf.…”

“…In the final part of the paper, we show that our ideas can be used to boost the performance of (i) the Efficient-Pose model [10] for 6D object pose estimation and (ii) the DSAC* Tiny model [8] for visual localization. In the appendix, we present a further experiment for visual localization using the Pixloc model [46].…”

“…Our experiments build on DSAC* Tiny [8] and the code available at [9]. DSAC* is an end-to-end learning method.…”

“…In RGB-mode, it uses a CNN to predict dense 2D-3D correspondences which are then used to estimate the camera pose using differentiable RANSAC [6]. For more details we refer the reader to [8] and Appendix B.2, where we also explain modifications we made to the code. DSAC* Tiny is a small version of DSAC*, requiring only 4MB of memory.…”