End-to-End Learning of Deep Visual Representations for Image Retrieval

Abstract

While deep learning has become a key ingredient in the top performing methods for many computer vision tasks, it has failed so far to bring similar improvements to instance-level image retrieval. In this article, we argue that reasons for the underwhelming results of deep methods on image retrieval are threefold: i) noisy training data, ii) inappropriate deep architecture, and iii) suboptimal training procedure. We address all three issues. First, we leverage a large-scale but noisy landmark dataset and develop an automatic cleaning method that produces a suitable training set for deep retrieval. Second, we build on the recent R-MAC descriptor, show that it can be interpreted as a deep and differentiable architecture, and present improvements to enhance it. Last, we train this network with a siamese architecture that combines three streams with a triplet loss. At the end of the training process, the proposed architecture produces a global image representation in a single forward pass that is well suited for image retrieval. Extensive experiments show that our approach significantly outperforms previous retrieval approaches, including state-of-the-art methods based on costly local descriptor indexing and spatial verification. On Oxford 5k, Paris 6k and Holidays, we respectively report 94.7, 96.6, and 94.8 mean average precision. Our representations can also be heavily compressed using product quantization with little loss in accuracy. To ensure the reproducibility of our research we have also released the clean annotations of the dataset and our pretrained models: http://www.xrce.xerox. com/Deep-Image-Retrieval.

Co-occurrence of deep convolutional features for image search

“…Query expansion repeats the search process based on a new query composed by the top ranked results on the first query [29], [30], [8], [7]. We use two different query expansion methods in our experiments.…”

“…Other approaches have tried to fine-tune the CNN with training datasets related to test datasets [8]. These approaches improve results in particular datasets, but have the drawback of requiring training datasets with expensive annotations depending on a category of each test set.…”

Unified Vision‐Based Methodology for Simultaneous Concrete Defect Detection and Geolocalization

“…In a similar spirit, Gordo et al. () proposed a trainable deep architecture that was built on the R‐MAC descriptors (Tolias et al., ) and achieved promising results on several instance retrieval benchmarks.…”

“…They achieved the state-of-the-art results on several image-based geolocalization and image retrieval tasks. In a similar spirit, Gordo et al (2016) proposed a trainable deep architecture that was built on the R-MAC descriptors (Tolias et al, 2015) and achieved promising results on several instance retrieval benchmarks.…”

Generic and attribute-specific deep representations for maritime vessels

“…Compelling amount of research efforts [27][28][29][30] have been put on content-based image retrieval (CBIR) as volumes of image databases are dramatically growing. Particularly, vessel retrieval is another promising application, potentially required in a maritime security system, where a user would like to query a database with a vessel image and retrieve similar images.…”