Deep Factorised Inverse-Sketching

Pang, Kaiyue; Li, Da; Song, Jifei; Song, Yi-Zhe; Xiang, Tao; Hospedales, Timothy M.

doi:10.1007/978-3-030-01267-0_3

Cited by 14 publications

(8 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sketch generation based on variational model [6] can automatically produce novel stroke sequences. Sketch abstraction [4,29] can generate sparser sketch whose semantic still could be recognized correctly. The sketch completion task [5] is a sketch inpainting task in white canvas.…”

Section: Sketch Generationmentioning

confidence: 99%

Disentangled and controllable sketch creation based on disentangling the structure and color enhancement

Gao

Hui

et al. 2021

IET Image Processing

View full text Add to dashboard Cite

Existing sketch-based image processing methods include sketch recognition, sketch synthesis and sketch-based image retrieval. For sketch creation, a meaningful task is proposed namely disentangled and controllable sketch creation (DCSC) based on disentangling the structure and color enhancement. Specifically, as the first subtask, sketch structure enhancement (SSE) is used to enhance a non-professional sketch (NPS) and obtain a professional sketch (PS), which is a process denoted as NPS2PS. A data set named SketchMan is first provided, consisting of NPSs and PSs with various postures in different scenes. SSE is trained as a conditional image-to-image translation problem, and there are three models: direct sketch-to-sketch (SS), grayscale guided SS and contour guided SS. Multiple IOU metrics are proposed based on Corner Point Map (CPM), Straight Line Map (SLM) and Segmented Area Map (SAM). As the second subtask, sketch color enhancement (SCE) is trained as a two-stage framework containing a topology enhancement network (TE-Net) that maps a sketch to the corresponding grayscale domain and a color injection network (CI-Net) that injects the global color feature to the AdaIN residual blocks to perform adaptive sketch colorization. The TE-Net and CI-Net disentangle the topological and color features to perform more controllable and diverse SCE results. Experimental results demonstrate that our proposed methods are effective to address the challenging and meaningful DCSC task compared with other state-of-the-art methods. INTRODUCTIONSketches have been studied in image classification, generation and retrieval [1][2][3][4][5][6]. A group of human-drawn strokes are supposed to depict an recognizable object. There are some existing datasets such as QMUL Shoe-V2 dataset [7], TU-Berlin dataset [8] and QuickDraw [6]. These collected sketches often consist of non-professional strokes and lack high-quality visual performance. Professional drawings usually belong to experienced artists. We present some examples in Figure 1a where the painter [9] gives a new life to his son's simple drawings by supplementing more important details and improving the abstract strokes. It is more difficult for a machine to model the refinement from NPS with ambiguous semantics to PS. More andThis is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

Section: Sketch Generationmentioning

confidence: 99%

Disentangled and controllable sketch creation based on disentangling the structure and color enhancement

Gao

Hui

et al. 2021

IET Image Processing

View full text Add to dashboard Cite

show abstract

“…As a byproduct, our network can also provide a highquality freehand sketch generator G A for a given photo [54,74,30]. We run our sketch extractor on COCO objects (top two rows) and web images (bottom two rows) and display the results in Figure 8.…”

Section: Photo-to-sketch Synthesismentioning

confidence: 99%

Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis

Xiang¹,

Liu²,

Xiao³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we explore the open-domain sketch-tophoto translation, which aims to synthesize a realistic photo from a freehand sketch with its class label, even if the sketches of that class are missing in the training data. It is challenging due to the lack of training supervision and the large geometry distortion between the freehand sketch and photo domains. To synthesize the absent freehand sketches from photos, we propose a framework that jointly learns sketch-to-photo and photo-to-sketch generation. However, the generator trained from fake sketches might lead to unsatisfying results when dealing with sketches of missing classes, due to the domain gap between synthesized sketches and real ones. To alleviate this issue, we further propose a simple yet effective open-domain sampling and optimization strategy to "fool" the generator into treating fake sketches as real ones. Our method takes advantage of the learned sketch-to-photo and photo-to-sketch mapping of in-domain data and generalizes them to the open-domain classes. We validate our method on the Scribble and SketchyCOCO datasets. Compared with the recent competing methods, our approach shows impressive results in synthesizing realistic color, texture, and maintaining the geometric composition for various categories of open-domain sketches.

show abstract

“…Recently, based on deep learning techniques, some other interesting sketch based multi-modal tasks have also been studied, e.g., text-to-sketch generation [231], sketch-based photo classifier generation [17], sketch-to-text pictionary game [232], [233], sketch to photo contour transfer [234]. Similarly, the challenges of these tasks are also mainly due to the large domain gap between free-hand sketches and other modalities.…”

Section: Other Sketch-related Multi-modal Tasksmentioning

confidence: 99%

Deep Learning for Free-Hand Sketch: A Survey

Xu¹

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Free-hand sketches are highly hieroglyphic and illustrative, which have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly more popular. The prosperity of deep learning has also immensely promoted the research for the free-hand sketch. This paper presents a comprehensive survey of the free-hand sketch oriented deep learning techniques. The main contents of this survey include: (i) The intrinsic traits and domain-unique challenges of the free-hand sketch are discussed, to clarify the essential differences between free-hand sketch and other data modalities, e.g., natural photo. (ii) The development of the free-hand sketch community in the deep learning era is reviewed, by surveying the existing datasets, research topics, and the state-of-the-art methods via a detailed taxonomy. (iii) Moreover, the bottlenecks, open problems, and potential research directions of this community have also been discussed to promote the future works.

show abstract

Deep Factorised Inverse-Sketching

Cited by 14 publications

References 42 publications

Disentangled and controllable sketch creation based on disentangling the structure and color enhancement

Disentangled and controllable sketch creation based on disentangling the structure and color enhancement

Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis

Deep Learning for Free-Hand Sketch: A Survey

Contact Info

Product

Resources

About