Learning local feature descriptors with triplets and shallow convolutional neural networks

Abstract

Finding correspondences between images via local descriptors is one of the most extensively studied problems in computer vision due to the wide range of applications. Recently, end-to-end learnt descriptors [1,2,3] based on Convolutional Neural Network (CNN) architectures and training on large datasets have demonstrated to significantly outperform state of the art features. These works are focused on exploiting pairs of positive and negative patches to learn discriminative representations.Recent work on deep learning for learning feature embeddings examines the use of triplets of samples instead of pairs. In this paper we investigate the use of triplets in learning local feature descriptors with CNNs and we propose a novel in-triplet hard negative mining step to achieve a more effective training and better descriptors. Our method reaches state of the art results without the computational overhead typically associated with mining of negatives and with lower complexity of the network architecture. This is a significant advantage over previous CNN-based descriptors since makes our proposal suitable for practical problems involving large datasets.Learning with triplets involves training from samples of the form {a a a, p p p, n n n}, where a a a is the anchor, p p p is a positive example, which is a different sample of the same class as a a a, and n n n is a negative example, belonging to a different class than a a a. In our case, a a a and p p p are different viewpoints of the same physical point, and n n n comes from a different keypoint. The goal is to learn the embedding f (x x x) s.t. δ + = || f (a a a) − f (p p p)|| 2 is low (i.e., the network brings a a a and p p p close in the feature space) and δ − = || f (a a a) − f (n n n)|| 2 is high (i.e., the network pushes the descriptors of a a a and n n n far apart). With this aim, we examine two different loss functions for triplet based-learning: the margin ranking loss and the ratio loss. The margin ranking loss is defined aswhere µ is an arbitrarily set margin. It measures the violation of the ranking order of the embedded features inside the triplet, which should be δ − > δ + + µ. If that is not the case, then the network adjusts its weights to achieve this result. For its part, the ratio loss optimises the ratio distances within triplets. It learns embeddings such thatThe goal of this loss function is to force ( e δ + e δ + +e δ − ) 2 to 0, and ( e δ − e δ + +e δ − ) 2 to 1. There is no margin associated with this loss, and by definition we have 0 ≤λ ≤ 1 for all values of δ − , δ + . Fig. 1 illustrates both approaches and their loss surface. In λ (δ + , δ − ) the loss remains 0 until the margin is violated, and after that, there is a linear increase not upper bounded. In contrast,λ (δ + , δ − ) has a clear slope between the two loss levels, and the loss reaches a 1-valued plateau quickly when δ − > δ + .All previous proposals based on triplet based learning use only two of the possible three distances within each triplet, ignoring the distance δ − = || f (p p ...

HPatches: A Benchmark and Evaluation of Handcrafted and Learned Local Descriptors

“…While the Oxford dataset contains images that are all captured by a camera, Generated Matching dataset [14] is obtained by generating images using synthetic transformations, and contains 16 sequences of 48 images. However, the synthetic nature of the transformations does not model all noise that typically occurs in the capturing process, thus making this data less challenging than the Oxford data [4]. The DTU Robots dataset [1] contains real images of 3D objects, captured using a robotic arm in controlled laboratory conditions, which is suitable for certain application scenarios but of limited diversity in the data.…”

Binary Online Learned Descriptors

“…The interest in CNNs based descriptors started from results shown in [29] that the features from the last layer of a convolutional deep network trained on ImageNet can outperform SIFT even though the networks were not specifically optimized for such local representations. End-to-end learning of patch descriptors using Siamese networks and the hinge contrastive loss [30], [31], [32] has recently been re-attempted in several works [29], [33], [34], [35], [36] and consistent improvements were reported over the state of the art descriptors in terms of matching performance. However, their efficiency is still far behind the traditional engineered descriptors and further progress has to be made to make their applications possible.…”

Cross-Spectral Local Descriptors via Quadruplet Network

“…Our work is based on the recent success of the triplet network presented in [3], named PN-Net, but adapted to work with cross-spectral image pairs, where for each matching pair, there are two possible non-matching patches; one for each spectrum. Results show that our technique is useful for learning cross-spectral feature descriptors that can be used as drop-in replacements of SIFT-like features descriptors.…”