SketchyScene: Richly-Annotated Scene Sketches

Zhang, Changqing; Yu, Qian; Du, Ruofei; Mo, Haoran; Song, Yi-Zhe; Xiang, Tao; Gao, Chengying; Chen, Baoquan; Zhang, Hao

doi:10.1007/978-3-030-01267-0_26

Cited by 61 publications

(70 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different learning schemes, such as multiple kernel learning [19] or active learning [43], may be employed for performance improvement. Another line of traditional methods has also attempted to utilize additional cues for recognition, such as prior knowledge for domain-specific sketches [1,15,27,23,32,2] or object context for sketched scenes [47,48]. While progress has been made in sketch recognition, these methods still cannot robustly handle freehand sketches with large shape or style variations, especially those hastily drawn in dozens of seconds [8], struggling to achieve performance on par with human on existing benchmarks like the TU-Berlin benchmark [3].…”

Section: Related Workmentioning

confidence: 99%

Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

Zhang

Zheng

et al. 2021

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

Freehand sketching is a dynamic process where points are sequentially sampled and grouped as strokes for sketch acquisition on electronic devices. To recognize a sketched object, most existing methods discard such important temporal ordering and grouping information from human and simply rasterize sketches into binary images for classification. In this paper, we propose a novel singlebranch attentive network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the dynamics in sketches for recognition. Sketch-R2CNN takes as input only a vector sketch with grouped sequences of points, and uses an RNN for stroke attention estimation in the vector space and a CNN for 2D feature extraction in the pixel space respectively. To bridge the gap between these two spaces in neural networks, we propose a neural line rasterization module to convert the vector sketch along with the attention estimated by RNN into a bitmap image, which is subsequently consumed by CNN. The neural line rasterization module is designed in a differentiable way to yield a unified pipeline for end-to-end learning. We perform experiments on existing large-scale sketch recognition benchmarks and show that by exploiting the sketch dynamics with the attention mechanism, our method is more robust and achieves better performance than the state-of-the-art methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

Zhang

Zheng

et al. 2021

IEEE Trans. Visual. Comput. Graphics

Self Cite

View full text Add to dashboard Cite

show abstract

“…In our prior work [Zou et al 2018], we conducted a pilot study on category-level semantic segmentation of scene sketches. In this paper, we investigate the problem of instance-level segmentation in scene sketches.…”

Section: Image Segmentationmentioning

confidence: 99%

“…Schneider and Tuytelaars [2016] performed sketch segmentation by looking at salient geometrical features (such as T-junctions and X-junctions) via a Conditional Random Field (CRF) framework. Instead of studying single object recognition or part-level sketch segmentation, we conduct an exploratory study for scene-level parsing of sketches, by using the large-scale scene sketch dataset SketchyScene [Zou et al 2018].…”

Section: Sketch Understandingmentioning

confidence: 99%

“…To address these problems, in [Zou et al 2018], we have built a large-scale scene sketch dataset, called SketchyScene, and conducted initial studies on category-level semantic segmentation of scene sketches. In this paper, we investigate two research goals: 1) the capability of deep neural networks for scene sketch understanding, and 2) the potential applications based upon instance-level understanding.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Language-based colorization of scene sketches

Zhang

Gao

et al. 2019

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

It's a moonlit night. There is a moon in the sky. A house is in the middle. A car is in front of the house. Three trees are on the left of the house. Two trees are on the right back of the car. There is a road.It is a moonlit night. There is a yellow moon in the sky. An orange house with light gray roof is in the middle. A yellow car with blue window is in front of the house. Three green trees are on the left of the house. Two green trees are on the right back of the car. There is a yellow road. The sky is blue and all things are on green grass.It is a moonlit night. There is a purple moon in the sky. A yellow house with yellow roof is in the middle. A blue car with light gray window is in front of the house. Three yellow trees are on the left of the house. Two yellow trees are on the right back of the car. There is a light brown road. The sky is gray and all things are on yellow grass.It is a moonlit night. There is a yellow moon in the sky. A blue house with blue roof is in the middle. A red car with blue window is in front of the house. Three green trees are on the left of the house. Two green trees are on the right back of the car. There is a dark gray road. The sky is blue and all things are on green grass. Fig. 1. We present LUCSS, a language-based interactive colorization system for scene sketches. It takes advantage of both instance level segmentation and language models in a unified generative adversarial network, allowing users to accomplish different colorization goals in a form of language instructions. Left: input scene sketch and its content description automatically generated by LUCSS. Three right columns show the colorization results generated by LUCSS following three different instructions at the bottom. Texts underlined are user-specified, with the target colors highlighted in bold.Abstract. We introduce LUCSS, a language-based system for interactive colorization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It consists of three sequential modules. First, given a scene sketch, the segmentation module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text description with spatial relationships based on the instance-level segmentation results. Finally, the interactive colorization module allows users to edit the caption and produce colored images based on the altered caption. Our experiments show the effectiveness of our approach and the desirability of its components to alternative choices.

show abstract

“…Scene sketches are more abstract and complicated due to multiple objects and their interactions. In this paper, we propose a novel problem of scene sketch zero-shot learning (SSZSL), which is more challenging than scene sketch understanding [8,9] and single-object sketch zero-shot learning [10].…”

Section: Introductionmentioning

confidence: 99%

Deep Zero-Shot Learning for Scene Sketch

Xie

2019

2019 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

We introduce a novel problem of scene sketch zero-shot learning (SSZSL), which is a challenging task, since (i) different from photo, the gap between common semantic domain (e.g., word vector) and sketch is too huge to exploit common semantic knowledge as the bridge for knowledge transfer, and (ii) compared with single-object sketch, more expressive feature representation for scene sketch is required to accommodate its high-level of abstraction and complexity. To overcome these challenges, we propose a deep embedding model for scene sketch zero-shot learning. In particular, we propose the augmented semantic vector to conduct domain alignment by fusing multi-modal semantic knowledge (e.g., cartoon image, natural image, text description), and adopt attention-based network for scene sketch feature learning. Moreover, we propose a novel distance metric to improve the similarity measure during testing. Extensive experiments and ablation studies demonstrate the benefit of our sketch-specific design.

show abstract

SketchyScene: Richly-Annotated Scene Sketches

Cited by 61 publications

References 39 publications

Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

Language-based colorization of scene sketches

Deep Zero-Shot Learning for Scene Sketch

Contact Info

Product

Resources

About