Transformer-Guided Convolutional Neural Network for Cross-View Geolocalization

Abstract

Ground-to-aerial geolocalization refers to localizing a ground-level query image by matching it to a reference database of geo-tagged aerial imagery. This is very challenging due to the huge perspective differences in visual appearances and geometric configurations between these two views. In this work, we propose a novel Transformer-guided convolutional neural network (TransGCNN) architecture, which couples CNN-based local features with Transformer-based global representations for enhanced representation learning. Specifically, our TransGCNN consists of a CNN backbone extracting feature map from an input image and a Transformer head modeling global context from the CNN map. In particular, our Transformer head acts as a spatialaware importance generator to select salient CNN features as the final feature representation. Such a coupling procedure allows us to leverage a lightweight Transformer network to greatly enhance the discriminative capability of the embedded features. Furthermore, we design a dual-branch Transformer head network to combine image features from multi-scale windows in order to improve details of the global feature representation. Extensive experiments on popular benchmark datasets demonstrate that our model achieves top-1 accuracy of 94.12% and 84.92% on CVUSA and CVACT val, respectively, which outperforms the second-performing baseline with less than 50% parameters and almost 2× higher frame rate, therefore achieving a preferable accuracy-efficiency tradeoff.

Co-Visual Pattern-Augmented Generative Transformer Learning for Automobile Geo-Localization

“…Baselines: Cross-view geo-localization (CVGL) has garnered significant research interest, resulting in several impressive works emerging in the field. To demonstrate the superiority of our proposed method, we selected 17 strong baselines and state-of-theart methods in total, i.e., Workman et al [9], Vo et al [10], Zhai et al [71], Cross-View Matching Network (CVM-Net) [11], Liu et al [31], Regmi et al [12], Spatial-Aware Feature Aggregation network (SAFA) [23], Cross-View Feature Transport technique (CVFT) [24], Dynamic Similarity Matching network (DSM) [73], Toker et al [41], Layer-to-Layer Transformer (L2LTR) [14], Local Pattern Network (LPN) [26], Unit SAFA + Subtraction Attention Module (USAM) [74], LPN + USAM [74], pure transformer-based geo-localization (Trans-Geo) [13], Transformer-Guided Convolutional Neural Network (TransGCNN) [25], and LPN + Dynamic Weighted Decorrelation Regularization (DWDR) [27]. In particular, for omnidirectional comparison, we use their recommended settings for training.…”

“…Unet-like [70] architecture comprising an encoder and a decoder has been widely used in generative tasks recently. Existing sduties [25,42] demonstrate that the attention mechanism in a transformer is excellent at modeling global contextual information, and CNN excels at encoding local semantic information. With these properties in mind, we propose a novel generative module that owns Unet-like [70] architecture and combines multi-head self-attention and convolutional layers in parallel for mutual benefit.…”

“…Owing to the dissimilar imaging perspective between satellite and ground views, the appearance of content varies significantly, posing a substantial challenge in achieving cross-view localization. Nonetheless, researchers have made remarkable strides in devising Siamese-like networks that contain two distinct branches responsible for encoding each view independently [14,[20][21][22][23][24][25][26][27][28][29]. While the relationship between different views provides a significant impetus for cross-view localization, several challenges persist.…”