SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval

Xu, Peng; Huang, Yongye; Yuan, Tongtong; Pang, Kaiyue; Song, Yi-Zhe; Xiang, Tao; Hospedales, Timothy M.; Ma, Zhanyu; Guo, Jun

doi:10.1109/cvpr.2018.00844

Cited by 116 publications

(106 citation statements)

References 32 publications

Supporting

Mentioning

104

Contrasting

Order By: Relevance

“…images, rather than vector representations of sketched strokes. In our approach we adopt the a vector representation for sketches, building upon the SketchRNN variational auto-encoder of Eck et al previously applied to blend [14] and match [37] sketches with sketches. Here we adapt SketchRNN in a more general form for both our interactive search of photographs, and for generating search suggestions, training with the Quickdraw50M dataset [1].…”

Section: Related Workmentioning

confidence: 99%

LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search

Collomosse

Bui

Jin

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

LiveSketch is a novel algorithm for searching large image collections using hand-sketched queries. LiveSketch tackles the inherent ambiguity of sketch search by creating visual suggestions that augment the query as it is drawn, making query specification an iterative rather than one-shot process that helps disambiguate users' search intent. Our technical contributions are: a triplet convnet architecture that incorporates an RNN based variational autoencoder to search for images using vector (stroke-based) queries; real-time clustering to identify likely search intents (and so, targets within the search embedding); and the use of backpropagation from those targets to perturb the input stroke sequence, so suggesting alterations to the query in order to guide the search. We show improvements in accuracy and time-to-task over contemporary baselines using a 67M image corpus.

show abstract

Section: Related Workmentioning

confidence: 99%

LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search

Collomosse

Bui

Jin

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Similar to deep metric learning, deep hashing aims to learn a discriminative embedding to preserve the consistency with semantic similarity in binary features. Recently, many deep hashing methods [40,16,38,41,18,31,25,42] have been proposed to learn compact binary codes and retrieve the similar images in Hamming space. Benefiting from metric learning methods, some deep hashing methods [17,14,35] are established on contrastive embedding or triplet embedding.…”

Section: Hashing Learningmentioning

confidence: 99%

Signal-To-Noise Ratio: A Robust Distance Metric for Deep Metric Learning

Yuan

Deng

Tang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

Deep metric learning, which learns discriminative features to process image clustering and retrieval tasks, has attracted extensive attention in recent years. A number of deep metric learning methods, which ensure that similar examples are mapped close to each other and dissimilar examples are mapped farther apart, have been proposed to construct effective structures for loss functions and have shown promising results. In this paper, different from the approaches on learning the loss structures, we propose a robust SNR distance metric based on Signal-to-Noise Ratio (SNR) for measuring the similarity of image pairs for deep metric learning. By exploring the properties of our SNR distance metric from the view of geometry space and statistical theory, we analyze the properties of our metric and show that it can preserve the semantic similarity between image pairs, which well justify its suitability for deep metric learning. Compared with Euclidean distance metric, our SNR distance metric can further jointly reduce the intra-class distances and enlarge the inter-class distances for learned features. Leveraging our SNR distance metric, we propose Deep SNR-based Metric Learning (DSML) to generate discriminative feature embeddings. By extensive experiments on three widely adopted benchmarks, including CARS196, CUB200-2011 and CIFAR10, our DSML has shown its superiority over other state-of-the-art methods. Additionally, we extend our SNR distance metric to deep hashing learning, and conduct experiments on two benchmarks, including CIFAR10 and NUS-WIDE, to demonstrate the effectiveness and generality of our SNR distance metric.

show abstract

“…While a human has the innate ability to interpret drawing semantics, the vast capacity of expressiveness in sketches poses great perception challenges to machines. For better human-computer interactions, sketch analysis has been an active research topic in the computer vision and graphics fields, spanning a wide spectrum including sketch recognition [3,44,47], sketch segmentation [35,11,17,18], sketch-based retrieval [4,38,30,42] and modeling [26], etc. In this paper, we focus on developing a novel learning-based method for freehand sketch recognition.…”

Section: Introductionmentioning

confidence: 99%

“…This representation does not allow the state-of-the-art convolutional neural networks (CNNs) to easily distinguish which strokes are more important or which strokes can be ignored for better recognition [31]. Following the definition in [42], a vector sketch in our work refers to a sequence of strokes containing the points in the drawing order ( Fig. 1).…”

Section: Introductionmentioning

confidence: 99%

“…A vector sketch can be easily converted into a bitmap image through rasterization but not vice versa. Notably, vector sketches contain rich temporal ordering and grouping (i.e., strokes) information, which has been shown to be useful for learning more descriptive features [42]. However, these information cues are all discarded during the rasterization process for pixel images and thus inaccessible by subsequent recognition al-gorithms.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

Zhang

Zheng

et al. 2021

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Freehand sketching is a dynamic process where points are sequentially sampled and grouped as strokes for sketch acquisition on electronic devices. To recognize a sketched object, most existing methods discard such important temporal ordering and grouping information from human and simply rasterize sketches into binary images for classification. In this paper, we propose a novel singlebranch attentive network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the dynamics in sketches for recognition. Sketch-R2CNN takes as input only a vector sketch with grouped sequences of points, and uses an RNN for stroke attention estimation in the vector space and a CNN for 2D feature extraction in the pixel space respectively. To bridge the gap between these two spaces in neural networks, we propose a neural line rasterization module to convert the vector sketch along with the attention estimated by RNN into a bitmap image, which is subsequently consumed by CNN. The neural line rasterization module is designed in a differentiable way to yield a unified pipeline for end-to-end learning. We perform experiments on existing large-scale sketch recognition benchmarks and show that by exploiting the sketch dynamics with the attention mechanism, our method is more robust and achieves better performance than the state-of-the-art methods.

show abstract

SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval

Cited by 116 publications

References 32 publications

LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search

LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search

Signal-To-Noise Ratio: A Robust Distance Metric for Deep Metric Learning

Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

Contact Info

Product

Resources

About