D2-Net: A Trainable CNN for Joint Description and Detection of Local Features

Abstract

In this work we address the problem of finding reliable pixel-level correspondences under difficult imaging conditions. We propose an approach where a single convolutional neural network plays a dual role: It is simultaneously a dense feature descriptor and a feature detector. By postponing the detection to a later stage, the obtained keypoints are more stable than their traditional counterparts based on early detection of low-level structures. We show that this model can be trained using pixel correspondences extracted from readily available large-scale SfM reconstructions, without any further annotations. The proposed method obtains state-of-the-art performance on both the difficult Aachen Day-Night localization dataset and the InLoc indoor localization benchmark, as well as competitive performance on other benchmarks for image matching and 3D reconstruction. Figure 1: Examples of matches obtained by the D2-Net method. The proposed method can find image correspondences even under significant appearance differences caused by strong changes in illumination such as day-to-night, changes in depiction style or under image degradation caused by motion blur.ciently via (approximate) nearest neighbor search [37] and the Euclidean distance. Sparse features offer a memory efficient representation and thus enable approaches such as Structure-from-Motion (SfM) [21,53] or visual localization [26,47,58] to scale. The keypoint detector typically considers low-level image information such as corners [19] or blob-like structures [30,32]. As such, local features can

Challenging Deep Image Descriptors for Retrieval in Heterogeneous Iconographic Collections

“…This departs from the traditional detectthen-describe process by selecting points after describing them, but it is simple to train and has shown good results on standard benchmarks [18]. Note that D typically ranges from 512 to 2048 in the last layers of the CNN, hence the PCA reduction to D = 40 Dusmanu et al [6] expanded this work with a detect-and-describe approach where they enforce keypoint repeatability and descriptor robustness using a Structure-from-Motion dataset with correponding points on different images.…”

Beyond Cartesian Representations for Local Descriptors

“…SuperPoint [9] learns scale invariance at the descriptor level, which works for visual odometry but breaks in more generalized problems. D2-Net [10] focuses on difficult imaging conditions and relies on a single network for detection and description. R2D2 [31] ap- plies L2-Net convolutionally while penalizing repeatable but non-discriminative patches.…”

“…As a result, the quest for ever-improving local feature descriptors goes on [23, 5, 46, 42, 39, 12, 50, 38, 41, 28, 45, This research was partially funded by Google's Visual Positioning System, the Swiss National Science Foundation, the Natural Sciences and Engineering Research Council of Canada, and by Compute Canada. 19,25,15,24,10,31]. These methods all seek to achieve invariance to small changes in location, orientation, scale, perspective, and illumination, along with imaging artefacts and partial occlusions.…”

Cascaded Parallel Filtering for Memory-Efficient Image-Based Localization

“…Comprehensive experiments on challenging real-world datasets demonstrate the benefit of our method. Further improvements could be achieved by incorporating CNN-based feature descriptors [11] or hierarchical localization schemes [27].…”