Localizing and Orienting Street Views Using Overhead Imagery

Abstract

In this paper we aim to determine the location and orientation of a ground-level query image by matching to a reference database of overhead (e.g. satellite) images. For this task we collect a new dataset with one million pairs of street view and overhead images sampled from eleven U.S. cities. We explore several deep CNN architectures for crossdomain matching -Classification, Hybrid, Siamese, and Triplet networks. Classification and Hybrid architectures are accurate but slow since they allow only partial feature precomputation. We propose a new loss function which significantly improves the accuracy of Siamese and Triplet embedding networks while maintaining their applicability to large-scale retrieval tasks like image geolocalization. This image matching task is challenging not just because of the dramatic viewpoint difference between ground-level and overhead imagery but because the orientation (i.e. azimuth) of the street views is unknown making correspondence even more difficult. We examine several mechanisms to match in spite of this -training for rotation invariance, sampling possible rotations at query time, and explicitly predicting relative rotation of ground and overhead images with our deep networks. It turns out that explicit orientation supervision also improves location prediction accuracy. Our best performing architectures are roughly 2.5 times as accurate as the commonly used Siamese network baseline.

Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources

“…This is very important for photo-sharing platforms, for which only a fraction of the uploaded photos comes with geo-location. Authors of [151,152] worked towards this aim, by training a cross-view…”

Generalization in Metric Learning: Should the Embedding Layer Be Embedding Layer?

“…While DML loss often involves looking at pair or triplet as the example, recent works construct a batch of images as input instead, and only form pairs/triplets/clusters at the loss layer. Hence the mini-batch can be constructed randomly in any way as long as a lot of similar/dissimilar pairs/triplet can be formed (for example [33,27]).…”

Hardness-Aware Deep Metric Learning

“…They take advantage of deep neural networks [17,27,31,11] to construct a mapping from the data space to the embedding space so that the Euclidean distance in the embedding space can reflect the actual semantic distance between data points, i.e., a relatively large distance between inter-class samples and a relatively small distance between intra-class samples. Recently a variety of deep metric learning methods have been proposed and have demonstrated strong effectiveness in various tasks, such as image retrieval [30,23,19,5], person re-identification [26,37,48,2], and geo-localization [35,14,34]. Figure 1.…”