SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs

Liu, Fang; Deng, Xiaoming; Zhang, Changqing; Lai, Yu‐Kun; Chen, Keqi; Zuo, Ran; Ma, Chenyue; Liu, Yong-Jin; Wang, Hongan

doi:10.1109/tip.2022.3175403

Cited by 9 publications

(1 citation statement)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We apply their proposed SketchyScene dataset to scene-level SBIR task. We noticed that the contemporaneous work SceneSketcher (Liu et al 2020) and SceneSketcher-v2 (Liu et al 2022) also employed GCN-based methods to achieve scenelevel SBIR. The crucial difference is that our solution is the first attempt to get rid of laborious instance-level annotation and implement scene-level SBIR system using minimal image-level pairing information, while the SceneSketcher series are useless for this purpose.…”

Section: Scene-level Sketch Understandingmentioning

confidence: 99%

Scene-Level Sketch-Based Image Retrieval with Minimal Pairwise Supervision

Wang

et al. 2023

AAAI

View full text Add to dashboard Cite

The sketch-based image retrieval (SBIR) task has long been researched at the instance level, where both query sketches and candidate images are assumed to contain only one dominant object. This strong assumption constrains its application, especially with the increasingly popular intelligent terminals and human-computer interaction technology. In this work, a more general scene-level SBIR task is explored, where sketches and images can both contain multiple object instances. The new general task is extremely challenging due to several factors: (i) scene-level SBIR inherently shares sketch-specific difficulties with instance-level SBIR (e.g., sparsity, abstractness, and diversity), (ii) the cross-modal similarity is measured between two partially aligned domains (i.e., not all objects in images are drawn in scene sketches), and (iii) besides instance-level visual similarity, a more complex multi-dimensional scene-level feature matching problem is imposed (including appearance, semantics, layout, etc.). Addressing these challenges, a novel Conditional Graph Autoencoder model is proposed to deal with scene-level sketch-images retrieval. More importantly, the model can be trained with only pairwise supervision, which distinguishes our study from others in that elaborate instance-level annotations (for example, bounding boxes) are no longer required. Extensive experiments confirm the ability of our model to robustly retrieve multiple related objects at the scene level and exhibit superior performance beyond strong competitors.

show abstract

Section: Scene-level Sketch Understandingmentioning

confidence: 99%

Scene-Level Sketch-Based Image Retrieval with Minimal Pairwise Supervision

Wang

et al. 2023

AAAI

View full text Add to dashboard Cite

show abstract

Contour detection network for zero-shot sketch-based image retrieval

Zhang

et al. 2023

Complex Intell. Syst.

View full text Add to dashboard Cite

Zero-shot sketch-based image retrieval (ZS-SBIR) is a challenging task that involves searching natural images related to a given hand-drawn sketch under the zero-shot scene. The previous approach projected image and sketch features into a low-dimensional common space for retrieval, and used semantic features to transfer the knowledge of seen to unseen classes. However, it is not effective enough to align multimodal features when projecting them into a common space, since the styles and contents of sketches and natural images are different and they are not one-to-one correspondence. To solve this problem, we propose a novel three-branch joint training network with contour detection network (called CDNNet) for the ZS-SBIR task, which uses contour maps as a bridge to align sketches and natural images to alleviate the domain gap. Specifically, we use semantic metrics to constrain the relationship between contour images and natural images and between contour images and sketches, so that natural image and sketch features can be aligned in the common space. Meanwhile, we further employ second-order attention to capture target subject information to increase the performance of retrieval descriptors. In addition, we use a teacher model and word embedding method to transfer the knowledge of the seen to the unseen classes. Extensive experiments on two large-scale datasets demonstrate that our proposed approach outperforms state-of-the-art CNN-based models: it improves by 2.6% on the Sketchy and 1.2% on TU-Berlin datasets in terms of mAP.

show abstract