Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization

Toker, Aysim; Zhou, Qunjie; Maximov, Maxim; Leal-Taixé, Laura

doi:10.1109/cvpr46437.2021.00642

Cited by 111 publications

(69 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2D ground-to-satellite image-based localization. Many recent works [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [38], [39], [40], [41], [42], [43], [44], [45] resort to satellite images as a reference set for image-based camera localization, due to the wide-spread coverage and easy accessibility of satellite imagery. Challenges of ground-to-satellite image matching include the significant visual appearance differences, geometric projection differences, and the unknown relative orientation between the two view images, as well as the limited FoV of query ground images.…”

Section: Related Workmentioning

confidence: 99%

“…Challenges of ground-to-satellite image matching include the significant visual appearance differences, geometric projection differences, and the unknown relative orientation between the two view images, as well as the limited FoV of query ground images. Existing works have focused on designing powerful network architectures [1], [3], [4], [7], [41], [43], bridging the cross-view domain gaps [6], [8], [9], [10], [45], and learning orientation invariant or equivariant features [4], [5], [10], [43], [44].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Accurate 3-DoF Camera Geo-Localization via Ground-to-Satellite Image Matching

Shi¹,

Yu²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

We address the problem of ground-to-satellite image geo-localization, that is, estimating the camera latitude, longitude and orientation (azimuth angle) by matching a query image captured at the ground level against a large-scale database with geotagged satellite images. Our prior arts treat the above task as pure image retrieval by selecting the most similar satellite reference image matching the ground-level query image. However, such an approach often produces coarse location estimates because the geotag of the retrieved satellite image only corresponds to the image center while the ground camera can be located at any point within the image. To further consolidate our prior research finding, we present a novel geometry-aware geo-localization method. Our new method is able to achieve the fine-grained location of a query image, up to pixel size precision of the satellite image, once its coarse location and orientation have been determined. Moreover, we propose a new geometry-aware image retrieval pipeline to improve the coarse localization accuracy. Apart from a polar transform in our conference work, this new pipeline also maps satellite image pixels to the ground-level plane in the ground-view via a geometry-constrained projective transform to emphasize informative regions, such as road structures, for cross-view geo-localization. Extensive quantitative and qualitative experiments demonstrate the effectiveness of our newly proposed framework. We also significantly improve the performance of coarse localization results compared to the state-of-the-art in terms of location recalls.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Accurate 3-DoF Camera Geo-Localization via Ground-to-Satellite Image Matching

Shi¹,

Yu²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…[Shi et al, 2020a] addressed the cross-view domain gap by applying a polar transform to the aerial images to approximately align the images up to an unknown azimuth angle. [Toker et al, 2021] proposed a GAN structure to create realistic street views from satellite images and localizes the corresponding query street-view simultaneously in an end-to-end manner. [Wang et al, 2021] proposed a square chunking strategy to fully extract useful information from the edges, and the chunking scheme results in a significant improvement in localization performance.…”

Section: Deeply-learned Feature For Geo-localizationmentioning

confidence: 99%

Vision-Based UAV Localization System in Denial Environments

Dai¹,

Huang²,

Zhuang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Unmanned Aerial Vehicle (UAV) localization capability is critical in a Global Navigation Satellite System (GNSS) denial environment. The aim of this paper is to investigate the problem of locating the UAV itself through a purely visual approach. This task mainly refers to: matching the corresponding geo-tagged satellite images through the images acquired by the camera when the UAV does not acquire GNSS signals, where the satellite images are the bridge between the UAV images and the location information. However, the sampling points of previous cross-view datasets based on UAVs are discrete in spatial distribution and the inter-class relationships are not established. In the actual process of UAV-localization, the inter-class feature similarity of the proximity position distribution should be small due to the continuity of UAV movement in space. In view of this, this paper have reformulated an intensive dataset for UAV positioning tasks, which is named DenseUAV, aiming to solve the problems caused by spatial distance and scale transformation in practical application scenarios, so as to achieve high-precision UAV-localization in GNSS denial environment. In addition, a new continuum-type evaluation metric named SDM is designed to evaluate the accuracy of model matching by exploiting the continuum of UAVs in space. Specifically, with the ideas of siamese networks and metric learning, a transformer-based baseline was constructed to enhance the capture of spatially subtle features. Ultimately, a neighbor-search post-processing strategy was proposed to solve the problem of large distance localisation bias.

show abstract

“…Meanwhile, GANs provide a feasible way to augment data in few-shot learning tasks [24]. As for autonomous driving, most methods focus on street view synthesis [25], [26], traffic sign generation [27], [28] and style transfer [29], [30]. Few works [31], [32] are designed for traffic lights.…”

Section: A Image and Video Generationmentioning

confidence: 99%

TL-GAN: Improving Traffic Light Recognition via Data Synthesis for Autonomous Driving

Wang¹,

Ma²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

Traffic light recognition, as a critical component of the perception module of self-driving vehicles, plays a vital role in the intelligent transportation systems. The prevalent deep learning based traffic light recognition methods heavily hinge on the large quantity and rich diversity of training data. However, it is quite challenging to collect data in various rare scenarios such as flashing, blackout or extreme weather, thus resulting in the imbalanced distribution of training data and consequently the degraded performance in recognizing rare classes. In this paper, we seek to improve traffic light recognition by leveraging data synthesis. Inspired by the generative adversarial networks (GANs), we propose a novel traffic light generation approach TL-GAN to synthesize the data of rare classes to improve traffic light recognition for autonomous driving. TL-GAN disentangles traffic light sequence generation into image synthesis and sequence assembling. In the image synthesis stage, our approach enables conditional generation to allow full control of the color of the generated traffic light images. In the sequence assembling stage, we design the style mixing and adaptive template to synthesize realistic and diverse traffic light sequences. Extensive experiments show that the proposed TL-GAN renders remarkable improvement over the baseline without using the generated data, leading to the state-of-the-art performance in comparison with the competing algorithms that are used for general image synthesis and data imbalance tackling.

show abstract

Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization

Cited by 111 publications

References 29 publications

Accurate 3-DoF Camera Geo-Localization via Ground-to-Satellite Image Matching

Accurate 3-DoF Camera Geo-Localization via Ground-to-Satellite Image Matching

Vision-Based UAV Localization System in Denial Environments

TL-GAN: Improving Traffic Light Recognition via Data Synthesis for Autonomous Driving

Contact Info

Product

Resources

About