Domain Adaptation of Networks for Camera Pose Estimation: Learning Camera Pose Estimation Without Pose Labels

Abstract

One of the key criticisms of deep learning is that large amounts of expensive and difficult-to-acquire training data are required in order to train models with high performance and good generalization capabilities. Focusing on the task of monocular camera pose estimation via scene coordinate regression (SCR), we describe a novel method, Domain Adaptation of Networks for Camera pose Estimation (DANCE), which enables the training of models without access to any labels on the target task. DANCE requires unlabeled images (without known poses, ordering, or scene coordinate labels) and a 3D representation of the space (e.g., a scanned point cloud), both of which can be captured with minimal effort using off-the-shelf commodity hardware. DANCE renders labeled synthetic images from the 3D model, and bridges the inevitable domain gap between synthetic and real images by applying unsupervised image-level domain adaptation techniques (unpaired image-to-image translation). When tested on real images, the SCR model trained with DANCE achieved comparable performance to its fully supervised counterpart (in both cases using PnP-RANSAC for final pose estimation) at a fraction of the cost. Our code and dataset are available at https://github.com/JackLangerman/dance * JL and ZQ contributed equally. This work was carried out while ZQ and DS interned at Nokia Bell Labs in 2020.

Reverse Domain Adaptation for Indoor Camera Pose Regression

“…Firstly, the reverse domain adaptation, i.e. the conversion of real images to synthetic has been explored for camera pose estimation, but using 'textured' 3D models derived from SfM (Yang et al, 2021a;Langerman et al, 2021;Yang et al, 2021b;Shoman et al, 2020). Therefore, the synthetic images used in those experiments, contain texture from the real world.…”

“…CycleGAN was utilised for increasing the number of synthetic images using a 3D textured model. Langerman et al (2021) propose the use of Contrastive Unpaired Translation (CUT) for performing unpaired image-image translation of synthetic images that are generated from coloured point clouds of the scene, and report their approach's performance is comparable with existing fully supervised techniques. Some works have reported that the real-synthetic strategy resulted in better performance for camera pose estimation.…”

A Common Knowledge-Driven Generic Vision Inspection Framework for Adaptation to Multiple Scenarios, Tasks, and Objects

“…Research in this field can be divided into direct estimation [24][25][26][27][28][29] and indirect estimation [30][31][32][33][34]. According to the supervision strategy, it can also be divided into fully supervised [24][25][26][27][28][30][31][32][33][34], semi-supervised [35,36], and unsupervised [37,38] directions. In the unsupervised direction, some emerging topics focus on few-shot learning [39], multi-modal learning [40], and virtual-real domain adaptation [41,42], etc.…”

“…As expressed in (10), it is also reasonable that the semantics of different patterns in the corresponding network layers should be similar. The domain adaptation performs the distribution alignment of a given pattern to the target pattern in the semantic space through technologies such as GAN [38]. This research can be seen as an extension of the former, targeting to replace manual embedding rules through end-to-end learning.…”