Improving Large-Scale Image Retrieval Through Robust Aggregation of Local Descriptors

Abstract

Abstract-Visual search and image retrieval underpin numerous applications, however the task is still challenging predominantly due to the variability of object appearance and ever increasing size of the databases, often exceeding billions of images. Prior art methods rely on aggregation of local scale-invariant descriptors, such as SIFT, via mechanisms including Bag of Visual Words (BoW), Vector of Locally Aggregated Descriptors (VLAD) and Fisher Vectors (FV). However, their performance is still short of what is required. This paper presents a novel method for deriving a compact and distinctive representation of image content called Robust Visual Descriptor with Whitening (RVD-W). It significantly advances the state of the art and delivers world-class performance. In our approach local descriptors are rank-assigned to multiple clusters. Residual vectors are then computed in each cluster, normalized using a direction-preserving normalization function and aggregated based on the neighborhood rank. Importantly, the residual vectors are de-correlated and whitened in each cluster before aggregation, leading to a balanced energy distribution in each dimension and significantly improved performance. We also propose a new post-PCA normalization approach which improves separability between the matching and non-matching global descriptors. This new normalization benefits not only our RVD-W descriptor but also improves existing approaches based on FV and VLAD aggregation. Furthermore, we show that the aggregation framework developed using hand-crafted SIFT features also performs exceptionally well with Convolutional Neural Network (CNN) based features. The RVD-W pipeline outperforms state-of-the-art global descriptors on both the Holidays and Oxford datasets. On the large scale datasets, Holidays1M and Oxford1M, SIFT-based RVD-W representation obtains a mAP of 45.1% and 35.1%, while CNN-based RVD-W achieve a mAP of 63.5% and 44.8%, all yielding superior performance to the state-of-the-art.

SIFT Meets CNN: A Decade Survey of Instance Retrieval

“…Concurrent to [54], Delhumeau et al [55] propose to normalize each residual vector instead of the residual sums; they also advocate for local PCA within each Voronoi cell which does not perform dimension reduction as [52]. A recent work [56] employs soft assignment and empirically learns optimal weights for each rank to improve over the hard quantization.…”

“…From these results, we arrive at three observations. First, among the SIFT-based methods, those with mediumsized codebooks [13], [31], [19] usually lead to superior (or competitive) performance, while those based on small codebook (compact representations) [15], [18], [56] exhibit inferior accuracy. On the one hand, the visual words in the medium-sized codebooks lead to relatively high matching recall due to the large Voronoi cells.…”

REMAP: Multi-Layer Entropy-Guided Pooling of Dense CNN Features for Image Retrieval

“…Virtually all aggregation schemes rely on clustering in feature space, with varying degree of sophistication: Bag-of-Words (BOW) [16], Vector of Locally Aggregated Descriptors (VLAD) [17], Fisher Vector (FV) [4], and Robust Visual Descriptor (RVD) [18]. BOW is effectively a fixed length histogram with descriptors assigned to the closest visual word; VLAD additionally encodes the positions of local descriptors within each voronoi region by computing their residuals; the Fisher Vector (FV) aggregates local descriptors using the Fisher Kernel framework (second order statistics), and RVD combines rank-based multi-assignment with robust accumulation to reduce the impact of outliers.…”

“…Kalantidis et al [22] extended this work by introducing cross-dimensional weighting in aggregation of CNN features. The retrieval performance is further improved when the RVD-W method is used for aggregation of CNNbased deep descriptors [18]. Tolias et al [2] proposed to extract Maximum Activations of Convolutions (MAC) descriptor from several multi-scale overlapping regions of the last convolutional layer feature map.…”

“…Retrieval accuracy is measured by mean Average Precision (mAP), as defined in [11]. To evaluate model retrieval accuracy in a more challenging scenario, the Holidays dataset is combined with 1 million distractor images obtained from Flickr, forming Holidays1M [18].…”