Filter Early, Match Late: Improving Network-Based Visual Place Recognition

Abstract

CNNs have excelled at performing place recognition over time, particularly when the neural network is optimized for localization in the current environmental conditions. In this paper we investigate the concept of feature map filtering, where, rather than using all the activations within a convolutional tensor, only the most useful activations are used. Since specific feature maps encode different visual features, the objective is to remove feature maps that are detract from the ability to recognize a location across appearance changes.Our key innovation is to filter the feature maps in an early convolutional layer, but then continue to run the network and extract a feature vector using a later layer in the same network. By filtering early visual features and extracting a feature vector from a higher, more viewpoint invariant later layer, we demonstrate improved condition and viewpoint invariance. Our approach requires image pairs for training from the deployment environment, but we show that state-of-the-art performance can regularly be achieved with as little as a single training image pair. An exhaustive experimental analysis is performed to determine the full scope of causality between early layer filtering and late layer extraction. For validity, we use three datasets: Oxford RobotCar, Nordland, and Gardens Point, achieving overall superior performance to NetVLAD. The work provides a number of new avenues for exploring CNN optimizations, without full re-training.

A visual place recognition approach using learnable feature map filtering and graph attention networks

“…Zhou et al [48] evaluated the performance of individual feature maps on various pixel-wise semantic segmentation tasks and found that a single feature map sensitive to different objects, textures, materials, and scenes. Besides, similar phenomena were also shown in the literature [18,16] we have mentioned before.…”

“…Unlike the off-line statistical methods used in [18,16], we follow SRM [20] to re-evaluate the weight of the channel and select the important feature maps in an end-to-end manner. In the FMF module as shown in Fig.…”

“…However, most of the ex-isting approaches have put so much emphasis on the image itself that they have ignored the architecture design of the network, where different filters may carry more meaningful information. Although several works like [18,16] revealed that some activations in intermediate layers are associated with environmental changes and tried to only extract the most useful feature maps for place recognition, prior off-line statistical processing was required. In this paper, we focus on filtering feature maps in an end-to-end manner with the filters learned by back-propagation.…”

Explicit feature disentanglement for visual place recognition across appearance changes

“…11 To go further, different supervised methods are proposed, which constrain the extracted features of images from the same place to be similar. 9,10,[32][33][34][35] One way is to use a classification network for place recognition where each place is a class, and this method can achieve comparable results. 9 The network is trained and tested on a dataset captured from static cameras at different times.…”

Hierarchical Multi-Process Fusion for Visual Place Recognition

“…In VPR specifically, [18] used a Greedy algorithm to find the optimal set of feature maps to use to improve localization performance, and further work used mutual information to determine the ideal subset of feature maps to use [19]. A recent work [20] intelligently selected the optimal reference set of images to use in VPR, by using Bayesian Selective Fusion.…”