Towards Practical Sketch-Based 3D Shape Generation: The Role of Professional Sketches

Abstract

In this paper, for the first time, we investigate the considerably more time to draw) and costly process (reward 83 per sketch is higher). 84 Even with the dataset, the problem of professional sketch 85 to 3D shape generation is non-trivial. There are two unique 86 difficulties brought by professional sketch that need to be 87 addressed. First, sketches naturally exhibit figure/ground am-88 biguity, i.e., the same sketch can lead to different 3D shape 89 interpretations (Fig. 2(a) offers examples). This ambiguity is 90 largely caused by foreground being hard to distinguish relying 91 solely on a few lines in absence of colour and texture. Second, 92 despite the sketches are professionally produced, misalignment 93 still exists between sketch and 3D shape as a result of different 94 drawing skills and styles of the artists. 95 As the second contribution, we design a deep adversarial 96 network to specifically tackle these challenges, where (i) a 97 discrimination-attention mechanism is developed to help fig-98 ure/ground estimation; we re-purpose the self-attention mech-99 anism in [15] and introduce a novel attention loss to ensure 100 that an automatically generated attention map aligns with that 101 of the ground-truth 3D shape, and (ii) in order to tackle the 102 inherent sketch-3D shape misalignment, we learn a global non-103 linear geometric transformation between an input sketch and 104 its 3D shape counterpart via a spatial transform network. Our 105 method builds on a recent work [4], where an adversarial 106 learning strategy is adopted to train a conditional GAN, which 107 is able to generate 2D representations of surface normal and 108 depth maps describing a 3D shape from different viewpoints, 109 which can then be fused into a full 3D representation. 110 Our contribution is summarized as follows: 111 • We contribute the first professional sketch and 3D shape 112 dataset, that contains 500 3D shapes and 1,500 profes-113 sional sketches, to drive future research.

SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling

“…Sketch-based modeling is a problem that has been studied for a long time. The earlier methods predicted local geometric properties from hand-crafted rules and then inferred the 3D shape from the geometric properties [33,32]. In recent years, some deep learning based methods have been proposed for sketch-based 3D modeling.…”

“…The work [32] discussed the additional challenges of line drawings in comparison with images in 3D reconstruction. In [33], sketches from two viewpoints were used as the input to perform 3D reconstruction, and they collected two datasets, ProSketch and AmateurSketch. Sketch2model [31] alleviated the ambiguity in sketch modeling by decoupling view code and shape code.…”

“…All sketch images are drawn by volunteers with different drawing skills. -AmateurSketch [33] contains 3, 015 sketch images of 500 chair models and 1,665 sketch images of 555 lamp models. Each 3D model is drawn from 3 different viewpoints.…”

“…Each 3D model is drawn from 3 different viewpoints. -ProSketch-3DChair [33] contains 1, 500 professional sketches of 500 chair models, and each 3D model is drawn from 3 different viewpoints: front, side and 45 degree.…”

SketchCleanNet -- A deep learning approach to the enhancement and correction of query sketches for a 3D CAD model retrieval system

“…The dataset, however, contains only a limited number of sketches (107) across 12 categories. The ProSketch3D [45] consists of 1500 sketches of 3D models across 500 object categories, taken from ShapeNet [46].…”