SFSegNet: Parse Freehand Sketches using Deep Fully Convolutional Networks

Jiang, Junkun; Wang, Ruomei; Lin, Sheng; Wang, Fei

doi:10.1109/ijcnn.2019.8851974

Cited by 8 publications

(5 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, recent years have seen the widespread adoption of deep learning [9,38] in SBIR and SBRSIR [25,26], outperforming hand-crafted methods [39]. Various deep network structures, including FCN [40], CNN [41], RNN [28], VAE [42], GNN [43], transformers [44], and ViT [13], have been explored. The latest solutions often combine multiple deep structures [13,41] and employ homogeneous [20], Siamese branch [9], or heterogeneous structures [21] for different modalities.…”

Section: Cross-model Feature Extractionmentioning

confidence: 99%

Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization

Yang,

Wang,

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

Effectively and efficiently retrieving images from remote-sensing databases is a critical challenge in the realm of remote-sensing big data. Utilizing hand-drawn sketches as retrieval inputs offers intuitive and user-friendly advantages, yet the potential of multi-level feature integration from sketches remains underexplored, leading to suboptimal retrieval performance. To address this gap, our study introduces a novel zero-shot, sketch-based retrieval method for remote-sensing images, leveraging multi-level feature extraction, self-attention-guided tokenization and filtering, and cross-modality attention update. This approach employs only vision information and does not require semantic knowledge concerning the sketch and image. It starts by employing multi-level self-attention guided feature extraction to tokenize the query sketches, as well as self-attention feature extraction to tokenize the candidate images. It then employs cross-attention mechanisms to establish token correspondence between these two modalities, facilitating the computation of sketch-to-image similarity. Our method significantly outperforms existing sketch-based remote-sensing image retrieval techniques, as evidenced by tests on multiple datasets. Notably, it also exhibits robust zero-shot learning capabilities in handling unseen categories and strong domain adaptation capabilities in handling unseen novel remote-sensing data. The method’s scalability can be further enhanced by the pre-calculation of retrieval tokens for all candidate images in a database. This research underscores the significant potential of multi-level, attention-guided tokenization in cross-modal remote-sensing image retrieval. For broader accessibility and research facilitation, we have made the code and dataset used in this study publicly available online.

show abstract

Section: Cross-model Feature Extractionmentioning

confidence: 99%

Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization

Yang,

Wang,

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…However, as shown in Figure 10, the goal is to perform pixel-wise segmentation of the semantic regions defined by the sketch, rather than the sketch strokes as in SSS. Existing models for sketch parsing thus far only use CNN-base networks to represent sketch the, e.g., SFSegNet [169] uses Deep Fully Convolutional Networks (FCN) [172].…”

Section: Sketch Parsingmentioning

confidence: 99%

Deep Learning for Free-Hand Sketch: A Survey

Xu¹,

Hospedales²,

Yin³

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Free-hand sketches are highly illustrative, and have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly popular. The progress of deep learning has immensely benefited free-hand sketch research and applications. This paper presents a comprehensive survey of the deep learning techniques oriented at free-hand sketch data, and the applications that they enable. The main contents of this survey include: (i) A discussion of the intrinsic traits and unique challenges of free-hand sketch, to highlight the essential differences between sketch data and other data modalities, e.g., natural photos. (ii) A review of the developments of free-hand sketch research in the deep learning era, by surveying existing datasets, research topics, and the state-of-the-art methods through a detailed taxonomy and experimental evaluation. (iii) Promotion of future work via a discussion of bottlenecks, open problems, and potential research directions for the community.

show abstract

“…In recent years, a new deep learning concept "sketch parse" [58], [120], [121], [122] appears in free-hand sketch community. As a kind of fine-grained semantic understanding of sketch, sketch parse has already been applied to assist other sketch tasks [58], e.g., sketch-based image retrieval (SBIR).…”

Section: Sketch Parsementioning

confidence: 99%

“…In particular, as shown in Figure 10, completely different to the stroke-level parse of sketch semantic segmentation models, sketch parse models work in the natural photo semantic segmentation manner, which parse sketches in area-level/part-level/patch-level ignoring the stroke traits of sketches thoroughly. The existing sketch parse models use only CNN-base architecture to represent sketch, e.g., SFSegNet [120] using Deep Fully Convolutional Networks (FCN) [123].…”

Section: Sketch Parsementioning

confidence: 99%

Deep Learning for Free-Hand Sketch: A Survey

Xu¹

2020

Preprint

View full text Add to dashboard Cite

Free-hand sketches are highly hieroglyphic and illustrative, which have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly more popular. The prosperity of deep learning has also immensely promoted the research for the free-hand sketch. This paper presents a comprehensive survey of the free-hand sketch oriented deep learning techniques. The main contents of this survey include: (i) The intrinsic traits and domain-unique challenges of the free-hand sketch are discussed, to clarify the essential differences between free-hand sketch and other data modalities, e.g., natural photo. (ii) The development of the free-hand sketch community in the deep learning era is reviewed, by surveying the existing datasets, research topics, and the state-of-the-art methods via a detailed taxonomy. (iii) Moreover, the bottlenecks, open problems, and potential research directions of this community have also been discussed to promote the future works.

show abstract

SFSegNet: Parse Freehand Sketches using Deep Fully Convolutional Networks

Cited by 8 publications

References 28 publications

Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization

Zero-Shot Sketch-Based Remote-Sensing Image Retrieval Based on Multi-Level and Attention-Guided Tokenization

Deep Learning for Free-Hand Sketch: A Survey

Deep Learning for Free-Hand Sketch: A Survey

Contact Info

Product

Resources

About