Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-Based Image Retrieval

Abstract

Low-shot sketch-based image retrieval is an emerging task in computer vision, allowing to retrieve natural images relevant to hand-drawn sketch queries that are rarely seen during the training phase. Related prior works either require aligned sketch-image pairs that are costly to obtain or inefficient memory fusion layer for mapping the visual information to a semantic space. In this paper, we address any-shot, i.e. zero-shot and few-shot, sketch-based image retrieval (SBIR) tasks, where we introduce the few-shot setting for SBIR. For solving these tasks, we propose a semantically aligned paired cycle-consistent generative adversarial network (SEM-PCYC) for any-shot SBIR, where each branch of the generative adversarial network maps the visual information from sketch and image to a common semantic space via adversarial training. Each of these branches maintains cycle consistency that only requires supervision at the category level, and avoids the need of aligned sketch-image pairs. A classification criteria on the generators’ outputs ensures the visual to semantic space mapping to be class-specific. Furthermore, we propose to combine textual and hierarchical side information via an auto-encoder that selects discriminating side information within a same end-to-end model. Our results demonstrate a significant boost in any-shot SBIR performance over the state-of-the-art on the extended version of the challenging Sketchy, TU-Berlin and QuickDraw datasets.

BDA-SketRet: Bi-Level Domain Adaptation for Zero-Shot SBIR

“…A key challenge in solving ZS-SBIR is because of the fact that sketches are usually drawn by various artists. ZS-SBIR (Yelamarthi et al, 2018;Dutta and Akata, 2020) models need to overcome a substantial within-category variance in addition to the domain gap between sketches and images given their disparity in spectral, spatial, and texture properties (Xian et al, 2017(Xian et al, , 2018Romera-Paredes and Torr, 2015). Moreover, natural images have random background effects which are completely absent in the sketches.…”

“…Moreover, natural images have random background effects which are completely absent in the sketches. In this regard, the majority of the existing ZS-SBIR approaches learn independent mappings to align the visual domains either in a semantic space (Dutta and Akata, 2019;Pandey et al, 2020) or in a semantics influ-enced latent space (Dey et al, 2019). In both cases, the representations are extracted from the entire image / sketch data while largely neglecting the complex distributions of different local regions.…”

“…This dilemma can be curtailed if the global feature extraction is guided by highly domain-independent mid-level regional properties. The existing work in the ZS-SBIR literature that use domain adaptation primarily tries to align the final features of the sketch and images (Dutta and Akata, 2019;Dey et al, 2019;. This often leads to a sub-optimal domain alignment due to variations in the sketch domain.…”

“…The zero-shot learning (ZSL) (Xian et al, 2017(Xian et al, , 2018Romera-Paredes and Torr, 2015) aims to bridge the gap between the non-overlapping sets of training classes (seen) and test classes (unseen) using semantic side information for visual recognition. To this end, attempts have been made to integrate ZSL with SBIR to get the zero-shot sketchbased image retrieval (ZS-SBIR) problem (Yelamarthi et al, 2018), even without constraining the search space to contain only the unseen classes at test time, e.g., generalized ZS-SBIR (GZS-SBIR) (Dutta and Akata, 2020). This essentially requires aligning the visual and semantic features (which is used for describing the class) in the embedding space.…”

Zero-Shot Sketch Based Image Retrieval using Graph Transformer

“…While the above-mentioned frameworks try to improve on the accuracy of the model by synthesizing a real-valued feature space, models such as [9], [16], [19] propose a hashed feature space which makes the retrieval a highly efficient process. While most of the existing frameworks demonstrate their results on ZS-SBIR, only a few report their performance on the more realistic task of GZS-SBIR [9], [13], [20].…”