Zero-Shot Sketch-Based Image Retrieval via Graph Convolution Network

Abstract

Zero-Shot Sketch-based Image Retrieval (ZS-SBIR) has been proposed recently, putting the traditional Sketch-based Image Retrieval (SBIR) under the setting of zero-shot learning. Dealing with both the challenges in SBIR and zero-shot learning makes it become a more difficult task. Previous works mainly focus on utilizing one kind of information, i.e., the visual information or the semantic information. In this paper, we propose a SketchGCN model utilizing the graph convolution network, which simultaneously considers both the visual information and the semantic information. Thus, our model can effectively narrow the domain gap and transfer the knowledge. Furthermore, we generate the semantic information from the visual information using a Conditional Variational Autoencoder rather than only map them back from the visual space to the semantic space, which enhances the generalization ability of our model. Besides, feature loss, classification loss, and semantic loss are introduced to optimize our proposed SketchGCN model. Our model gets a good performance on the challenging Sketchy and TU-Berlin datasets.

Norm-guided Adaptive Visual Embedding for Zero-Shot Sketch-Based Image Retrieval

“…Most existing ZS-SBIR methods follow the common space paradigm and pre-define class-prototypes with language models. Graph [Zhang et al, 2020;Shen et al, 2018], cycle consistency [Dutta and Akata, 2019;Deng et al, 2020] and content-style disentanglement [Dutta and Biswas, 2019]) are employed to learn the projection to map sketches/photos close to such prototypes. [Dutta and Akata, 2019] proposes a selection layer to refine the prototypes and reduce the dimensionality of retrieval features.…”

“…[Dutta and Akata, 2019] proposes a selection layer to refine the prototypes and reduce the dimensionality of retrieval features. Graph-based method [Zhang et al, 2020] adjust the language-based adjacency matrix with visual information. proposes a teacherstudent framework to preserve discriminative representations from ImageNet and coordinates the representations close to language-based prototypes.…”

“…Then, they learn to project both sketches and photos close to their corresponding class-prototypes. Based on this paradigm, various architectures (e.g., GANs [Dutta and Biswas, 2019], graph [Shen et al, 2018;Zhang et al, 2020], and cycle reconstruction [Dutta and Akata, 2019;Deng et al, 2020]) have been employed to estimate the projection. They expect that such learned mappings can leverage the side-information from language model to project unseen class sketches/photos close to their semantically similar seen classes, and thus the distances among the sketches and photos from the same category are minimized.…”

Zero-Shot Sketch Based Image Retrieval using Graph Transformer

“…The utility of semantic space has been validated in various previously proposed frameworks, such as [9], [10], [19], [21], to name a few. The visual space is aligned with the semantic space to aid the training process and learn from the topology of the semantic space.…”

“…However, while [9], [10], [19] used the semantic space, they made the semantic space latent and learnable causing the network to eventually loose the classwise topology information as the network is trained for more number of epochs. To preserve the original topology of the semantic space, [21], [22] proposed using a graph convolution network (GCN) [23]. While in [22] the authors use a GCN directly on the semantic graph, in [21] the authors create a fully-connected graph whose edge weights correspond to the semantic distances and the node features comprise of classwise visual features.…”