Camera Relocalization by Computing Pairwise Relative Poses Using Convolutional Neural Network

Abstract

We propose a new deep learning based approach for camera relocalization. Our approach localizes a given query image by using a convolutional neural network (CNN) for first retrieving similar database images and then predicting the relative pose between the query and the database images, whose poses are known. The camera location for the query image is obtained via triangulation from two relative translation estimates using a RANSAC based approach. Each relative pose estimate provides a hypothesis for the camera orientation and they are fused in a second RANSAC scheme. The neural network is trained for relative pose estimation in an end-to-end manner using training image pairs. In contrast to previous work, our approach does not require scene-specific training of the network, which improves scalability, and it can also be applied to scenes which are not available during the training of the network. As another main contribution, we release a challenging indoor localisation dataset covering 5 different scenes registered to a common coordinate frame. We evaluate our approach using both our own dataset and the standard 7 Scenes benchmark. The results show that the proposed approach generalizes well to previously unseen scenes and compares favourably to other recent CNN-based methods.

Understanding the Limitations of CNN-Based Absolute Camera Pose Regression

“…The prediction is again handled by a CNN trained for regression. Relevant training images can be found using an explicit image retrieval step [7,35] or by implicitly representing the images in the CNN [56]. APR is an instancelevel problem, i.e., APR techniques need to be trained for a specific scene.…”

“…6 also establishes a relation between APR approaches and relative pose regression (RPR) algorithms. RPR methods first identify a set of training images relevant to a given test image, e.g., using image retrieval [7,35] or by encoding the training images in a CNN [56]. They then compute a pose offset from the training images to the test image via regression.…”

“…They can thus also be trained on unrelated scenes. Besides RelocNet and Relative PN trained on 7 Scenes (7S), we thus also compare against variants trained on other datasets (ScanNet (SN) [20], University (U) [35]). As shown in Tab.…”

RSL-Net: Localising in Satellite Images From a Radar on the Ground

“…Given training data from a previously seen environment, the work by Kendall et al [22,23] directly regresses the 6-DoF camera pose for relocalisation. For a live query image, Laskar et al [24] learned a descriptor to find matches from a database of prior images, and regressed the pairwise relative pose. The method presented in [25] solves camera relocalisation by learning 3D scene coordinates.…”

A survey of deep learning techniques for autonomous driving

“…, or the approaches introduced inWalch et al (2017),Melekhov, Ylioinas, Kannala, and Rahtu (2017),Laskar, Melekhov, Kalia, and Kannala (2017),Brachmann and Rother (2018), or Sarlin, Debraine, Dymczyk, Siegwart, andCadena (2018), are using image data to estimate the 3D pose of a camera in an End2End fashion. Scene semantics can be derived together with the estimated pose(Radwan et al, 2018).LiDAR intensity maps are also suited for learning a real-time, calibration-agnostic localization for autonomous cars (Barsan, Wang, Pokrovsky, & Urtasun, 2018).…”