Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression

Abstract

We propose and evaluate several deep network architectures for measuring the similarity between sketches and photographs, within the context of the sketch based image retrieval (SBIR) task. We study the ability of our networks to generalize across diverse object categories from limited training data, and explore in detail strategies for weight sharing, pre-processing, data augmentation and dimensionality reduction. In addition to a detailed comparative study of network configurations, we contribute by describing a hybrid multi-stage training network that exploits both contrastive and triplet networks to exceed state of the art performance on several SBIR benchmarks by a significant margin. Datasets and models are available at www.cvssp.org.

LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search

“…Our embedding (LS) outperforms all ablations and baselines, with vector query alone contributing significant margin over raster. The addition of fc layers to create cross-modal embedding (-R) slightly improves (importantly, does not degrade) the intermediate raster embedding R available via [6]. The method significantly outperforms recent triplet SBIR approaches [28,5].…”

“…LiveSketch accepts a query sketch Q in vector graphics form (as a variable length sequence of strokes), and searches a large (∼ 10 8 ) dataset of raster images I = {I 1 , ..., I N }. Our two-stream network architecture ( ; the image branch of [6]. Query sketches are encoded via SQ(.…”

“…1). A core novelty of our approach lies within the method by which visual suggestions are generated, exploiting the reversibility of deep neural networks (DNNs) that are commonly used to encode image features to create the search index in visual search systems [26,13,7,6]. By identifying clusters of likely target intents for the user's search, we reverse the DNN encoder to explain how such clusters could be gen- Figure 1.…”

“…Average time to retrieve 20 class-and instance-level search targets (18 participants, 3 per method). Comparing LiveSketch (LS) interactive method with ablations (-NI) non-interactive/one-shot; (-R) raster substitutes vector query; (-I) intermediate structure embedding, and with the three baselines[6,28,5]. Times in seconds; parentheses total the averaged missed queries.…”

A Semi-Supervised Approach to Sar-Optical Image Matching

“…This was achieved using a statistical dependency measure to pair unlabeled data during training and supervised with corresponding training pairs. Using a multi-phase training approach (Bui et al, 2018) pretrained a classifier for each domain in a supervised manner and then used a second training phase to learn a transformation between the learned embeddings for cross-domain image retrieval.…”