Gradient field descriptor for sketch based retrieval and localization

Abstract

We present an image retrieval system driven by free-hand sketched queries depicting shape. We introduce Gradient Field HoG (GF-HOG) as a depiction invariant image descriptor, encapsulating local spatial structure in the sketch and facilitating efficient codebook based retrieval. We show improved retrieval accuracy over 3 leading descriptors (Self Similarity, SIFT, HoG) across two datasets (Flickr160, ETHZ extended objects), and explain how GF-HOG can be combined with RANSAC to localize sketched objects within relevant images. We also demonstrate a prototype sketch driven photo montage application based on our system.

Visual Saliency Weighting and Cross-Domain Manifold Ranking for Sketch-Based Image Retrieval

“…We used two sketch-based image retrieval benchmark databases; the Flickr160 [6] and the Flickr15k [7], both by Hu et al Figure 4 shows examples of sketch queries and retrieval target images for the two benchmarks.…”

“…The CDMR improved retrieval accuracy very significantly. For example, for the Flickr160 benchmark, the combination of the VSW and the CDMR produced MAP score of 72.3 %, which is about 18 % higher than 54.0 % of the BF-GFHoG reported in [6].…”

“…We experimentally evaluated the proposed algorithm by using two sketch-based image retrieval benchmarks by Hu et al, that are, the Flickr160 [6] and the Flickr15k [7]. Small but consistent improvement in retrieval accuracy is observed for both benchmarks when the VSW is applied.…”

“…Saavedra et al [3] uses a set of local histograms of Canny edge orientations as an image feature, while Eitz et al [2] combines several orientation sensitive image features including a variant of HoG [14]. The BF-GFHoG by Hu et al [6] [7] employs gradient field image as the common ground and extracts HoG descriptors. Hundreds of HoG descriptors extracted from a gradient field image are integrated into a feature vector per image by using the Bag-of-Features (BF) approach.…”

“…A SBIR algorithm uses a common ground representation, either line/edge image [1][2][3][4][5] or gradient field image [6,7], for sketch-to-image comparison. Then, one or more image features are extracted from the common ground representation for comparison.…”