Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval

Abstract

While image retrieval and instance recognition techniques are progressing rapidly, there is a need for challenging datasets to accurately measure their performance -while posing novel challenges that are relevant for practical applications. We introduce the Google Landmarks Dataset v2 (GLDv2), a new benchmark for large-scale, fine-grained instance recognition and image retrieval in the domain of human-made and natural landmarks. GLDv2 is the largest such dataset to date by a large margin, including over 5M images and 200k distinct instance labels. Its test set consists of 118k images with ground truth annotations for both the retrieval and recognition tasks. The ground truth construction involved over 800 hours of human annotator work. Our new dataset has several challenging properties inspired by realworld applications that previous datasets did not consider: An extremely long-tailed class distribution, a large fraction of out-of-domain test photos and large intra-class variability. The dataset is sourced from Wikimedia Commons, the world's largest crowdsourced collection of landmark photos. We provide baseline results for both recognition and retrieval tasks based on state-of-the-art methods as well as competitive results from a public challenge. We further demonstrate the suitability of the dataset for transfer learning by showing that image embeddings trained on it achieve competitive retrieval performance on independent datasets. The dataset images, ground-truth and metric scoring code are available at https://github.com/cvdfoundation/google-landmark.

Are we done with ImageNet?

“…Related work. Several studies have revisited common computer vision benchmarks [16,17,18,19] and, regarding ImageNet [1] specifically, identified various sources of bias and noise [20,21,22]. However, none of these investigate the effect of ImageNet's shortcomings, in particular how they might affect model accuracies and conclusions being drawn from them.…”

Benchmarking Image Retrieval for Visual Localization

“…They learn the similarity metric by using ranking losses such as contrastive, triplet, or average precision (AP) [8,31,69,71]. Several benchmark papers compare such image representations on the task of instancelevel landmark retrieval [4,63,66,68,105,112]. In contrast, this paper explores how state-of-the-art landmark retrieval approaches perform in the context of visual localization.…”

Few-shot Long-Tailed Bird Audio Recognition

“…Therefore, physical monitoring is difficult, and scientists have turned to sound recordings. As we show in Figure 2, the distribution of the "interesting" bird species is very long-tailed [29], making it necessary to deal with extreme class imbalance. As we introduced, in this competition, our challenge is to develop ML models to identify bird species using sounds.…”