Semantic Cross-View Matching

Abstract

Matching cross-view images is challenging because the appearance and viewpoints are significantly different. While low-level features based on gradient orientations or filter responses can drastically vary with such changes in viewpoint, semantic information of images however shows an invariant characteristic in this respect. Consequently, semantically labeled regions can be used for performing cross-view matching.In this paper, we therefore explore this idea and propose an automatic method for detecting and representing the semantic information of an RGB image with the goal of performing cross-view matching with a (non-RGB) geographic information system (GIS). A segmented image forms the input to our system with segments assigned to semantic concepts such as traffic signs, lakes, roads, foliage, etc. We design a descriptor to robustly capture both, the presence of semantic concepts and the spatial layout of those segments. Pairwise distances between the descriptors extracted from the GIS map and the query image are then used to generate a shortlist of the most promising locations with similar semantic concepts in a consistent spatial layout. An experimental evaluation with challenging query images and a large urban area shows promising results.

Semantic Visual Localization

“…Aerial-Ground Localization. A related problem to ours is tackled by work on matching ground-level imagery against overhead maps to obtain coarse location estimates under orthogonal viewpoint changes [13,40,71,73]. However, these methods are specific to this problem and cannot be used for accurate ground-level to ground-level localization, which is the focus of this paper.…”

Find your way by observing the sun and other semantic cues

“…However, additional sensors are required for this method, while our approach directly reasons about the sun direction from images. Another interesting work exploited semantic labeling from a single image and matching with the GIS dataset [5]. However, using semantics from a single image alone cannot achieve a meter-level accuracy in large-scale urban environments, where the semantic layout of the scene is very repetitive.…”

A survey on Visual-Based Localization: On the benefit of heterogeneous data

“…However, finding similarity between data acquired at a ground level and data captured with flying devices is a hard task due to the extreme change in viewpoint. A series of works consider cross-view localization [104,218,30,207,195]. In [218,207], authors investigate the use of a CNN to automatically associate ground level images taken from street view service with fine-grained overhead images.…”

“…Graph representation offers a compressed data representation capable to handle local changes and minor measurement errors [189]. Semantic segmentation is used in [112] to narrow the search scope and in [6,30,38] to directly recover the pose of the query (illustration on figure 3c). Works described in [5,132] consider the re-weighting of extracted local features in image according to the semantic class of the pixel obtained by image segmentation.…”

“…In a same manner, Arth et al [7] present concrete application of semantic segmentation of an image to reinforce hypothesis about building facades segmentation (the image is segmented with a SVM classifier). On the other hand, several methods rely on annotated map [9,215] or Geographic Information System (GIS) [6,30,159] to guide the localization.…”