SEKD: Self-Evolving Keypoint Detection and Description

Abstract

Researchers have attempted utilizing deep neural network (DNN) to learn novel local features from images inspired by its recent successes on a variety of vision tasks. However, existing DNN-based algorithms have not achieved such remarkable progress that could be partly attributed to insufficient utilization of the interactive characters between local feature detector and descriptor. To alleviate these difficulties, we emphasize two desired properties, i.e., repeatability and reliability, to simultaneously summarize the inherent and interactive characters of local feature detector and descriptor. Guided by these properties, a self-supervised framework, namely self-evolving keypoint detection and description (SEKD), is proposed to learn an advanced local feature model from unlabeled natural images. Additionally, to have performance guarantees, novel training strategies have also been dedicatedly designed to minimize the gap between the learned feature and its properties. We benchmark the proposed method on homography estimation, relative pose estimation, and structurefrom-motion tasks. Extensive experimental results demonstrate that the proposed method outperforms popular handcrafted and DNN-based methods by remarkable margins. Ablation studies also verify the effectiveness of each critical training strategy. We will release our code along with the trained model publicly.

ALIKE: Accurate and Lightweight Keypoint Detection and Descriptor Extraction

“…SuperPoint [11] first trains a MagicPoint model on synthetic dataset, and then bootstraps the score map on real images with homographic adaption strategy, and its descriptors are trained with triplet loss. This strategy is also adopted in MLIFeat [27] and SEKD [19]. R2D2 [12] identifies keypoints as reliable and repeatable positions in image and trains reliability through AP loss [31].…”

“…1). The widely use method to train sparse descriptors is the triplet loss [11], [13], [14], [19], but our experiments indicate that the training of sub-pixel sparse descriptors with triplet loss is tricky and unstable, as they only cover the sampled keypoints of entire descriptor map.…”

“…Latter, many excellent methods address the keypoint detection and descriptor extraction with a single network [11]- [14]. Some of them [11], [12], [15]- [19] simply treat the keypoint detection as a score map estimation problem, where the score of a pixel indicates the probability that it is a keypoint. And then they train the intermediate score map with the synthetic ground truth score map and/or the similarity of Compared with the Non-Maximum-Suppression (NMS) based methods, the DKD can back-propagate the gradients and produce sub-pixel keypoints.…”

“…We present the keypoint reprojection loss to train the sub-pixel keypoints detected from DKD, which directly minimizes the reprojection distance of detected keypoints between images. Inspired by the peak loss [12], [19], a dispersity peak loss is further introduced to avoid blob scores on score map, which forces the score map to be accurately "peaky" at the keypoint position.…”

“…However, existing methods pay more attention on the matching performance rather than running efficiency. To further improve the running efficiency, a lightweight convolutional neural network (CNN) is designed by simply concatenating multi-level features [14], [19], [27], for both localization accuracy and representation capabilities. The experiments indicate that this lightweight network has comparable performance to existing methods while runs much faster.…”