Learning to Detect Scene Landmarks for Camera Localization

Abstract

Camera localization methods based on retrieval, local feature matching, and 3D structure-based pose estimation are accurate but require high storage, are slow, and are not privacy-preserving. A method based on scene landmark detection (SLD) was recently proposed to address these limitations. It involves training a convolutional neural network (CNN) to detect a few predetermined, salient, scene-specific 3D points or landmarks and computing camera pose from the associated 2D-3D correspondences. Although SLD outperformed existing learning-based approaches, it was notably less accurate than 3D structure-based methods. In this paper, we show that the accuracy gap was due to insufficient model capacity and noisy labels during training. To mitigate the capacity issue, we propose to split the landmarks into subgroups and train a separate network for each subgroup. To generate better training labels, we propose using dense reconstructions to estimate visibility of scene landmarks. Finally, we present a compact architecture to improve memory efficiency. Accuracy wise, our approach is on par with state of the art structurebased methods on the INDOOR-6 dataset but runs significantly faster and uses less storage. Code and models can be found at https://github.com/microsoft/ SceneLandmarkLocalization.

How Privacy-Preserving are Line Clouds? Recovering Scene Details from 3D Lines

“…Approaches based on implicit representations map image pixels or patches to 3D points by training scene coordinate regression models [3,38]. Recently, it was claimed that such approaches are inherently privacy-preserving [11]. However, feature-based methods currently scale better to large scenes and are able to better handle condition changes [44], such as illumination or seasonal changes, between the query image and the database images used to build the the scene representation.…”

“…Work on privacy-preserving queries to a localization server thus mostly aims at developing features that prevent image recovery [14,26] or on obfuscating the feature geometry [16,42]. Similarly, work on privacy-preserving scene representation aims to obfuscate the geometry [37,41] (although scene geometry can be recovered under certain conditions [7]), splitting the maps over multiple server for increased data security [15], using implicit representations [11], or storing raw geometry without any feature descriptors [52].…”

“…Naturally, sending user data to a server, e.g., in the form of images to be localized or 3D maps recorded by users that will be used for localization, raises privacy concerns [9,41,42]. Work on privacy-preserving localization aims to resolve these concerns by ensuring that private details cannot be recovered from the data sent [14,26,42] to or stored on the server [11,11,15,28,36,41,52].…”

An Outlook into the Future of Egocentric Vision

“…Generally, algorithms for visual localisation mostly rely on complex 3D point clouds that are expensive to build, store, and maintain over time. Do et al (2022a) trained a CNN to detect the appearance of a sparse set of 3D scene points (scene landmarks), and showed that those predicted landmarks can yield accurate pose estimates, while being privacy preserving and requiring low data storage. Panek et al (2022) explored dense 3D scene models as an alternative to the sparse Structure-from-Motion point clouds as they are more flexible than SfM-based representations and can be rather compact.…”