Cross-view Geo-localization with Evolving Transformer

Abstract

In this work, we address the problem of cross-view geo-localization, which estimates the geospatial location of a street view image by matching it with a database of geo-tagged aerial images. The cross-view matching task is extremely challenging due to drastic appearance and geometry differences across views. Unlike existing methods that predominantly fall back on CNN, here we devise a novel evolving geo-localization Transformer (EgoTR) that utilizes the properties of self-attention in Transformer to model global dependencies, thus significantly decreasing visual ambiguities in cross-view geo-localization. We also exploit the positional encoding of Transformer to help the EgoTR understand and correspond geometric configurations between ground and aerial images. Compared to state-of-the-art methods that impose strong assumption on geometry knowledge, the EgoTR flexibly learns the positional embeddings through the training objective and hence becomes more practical in many real-world scenarios. Although Transformer is well suited to our task, its vanilla self-attention mechanism independently interacts within image patches in each layer, which overlooks correlations between layers. Instead, this paper propose a simple yet effective self-cross attention mechanism to improve the quality of learned representations. The self-cross attention models global dependencies between adjacent layers, which relates between image patches while modeling how features evolve in the previous layer. As a result, the proposed self-cross attention leads to more stable training, improves the generalization ability and encourages representations to keep evolving as the network goes deeper. Extensive experiments demonstrate that our EgoTR performs favorably against state-of-the-art methods on standard, fine-grained and cross-dataset cross-view geo-localization tasks. Codes are available in the supplementary material.

Multi-branch offset architecture for unaligned cross-view geo localization

“…In 2020, Dosovitskiy et al [9] creatively applied Transformer to the image field, divided continuous images into discrete Tokens through Patch Embedding, and converted the input into a sequence. In 2021, EgoTR [10] first introduced the Transformer model into the cross view localization task, enabling global context learning and location aware representation. At the same time, it also proposed a novel self-attention to promote cross layer information flow and achieve the transcendence of the CNN model.…”

“…The inference speed is 1.48 times that of DSM and 2.52 times that of EgoTR, reaching the real-time speed of 33.78 FPS. Param(MB) FPS(Frame per second) SAFA [6] 29.5 35.90 DSM [8] 17.9 22.72 EgoTR [10] 195.9 3, the offset module has N branches with identical structures but not shared parameters. Different branches are responsible for learning the features of different offsets.…”

A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localization

“…Tulder et al [57] presented a novel cross-view transformer method to transfer information between unregistered views at the level of spatial feature maps, which achieved remarkable results in field of Multi-view medical image analysis. Yang et al [58] proposed a simple yet effective self-cross attention mechanism to improve the quality of learned representations. Which improved the generalization ability and encourages representations to keep evolving as the network goes deeper.…”