3D Sketching using Multi-View Deep Volumetric Prediction

Delanoy, Johanna; Aubry, Mathieu; Isola, Phillip; Efros, Alexei A.; Bousseau, Adrien

doi:10.1145/3203197

Cited by 116 publications

(124 citation statements)

References 76 publications

Supporting

Mentioning

122

Contrasting

Order By: Relevance

“…16: Ablations study on V2VNet. The method by Delanoy et al [37] tends to generate vector fields whose values are inconsistent with the ones in the previous view and thus are often chaotic (the middle column). For each method, we show the corresponding results in the previous and current views.…”

Section: Methodsmentioning

confidence: 99%

“…Huang et al [36] introduce deep convolutional neural network (CNN) for mapping 2D sketches to procedural model parameters. Delanoy et al [37] propose an end-to-end CNN, which is trained to generate 3D models from 2D multiview sketches. In [30], Li et al learn 2D middle maps to guide robust 3D modeling from 2D sketches.…”

Section: Related Workmentioning

confidence: 99%

“…In this process, we hope that our network only changes the 3D orientation field in the new view with respect to the new sketch, while it can keep the original values in the other areas. Existing works on multi-view modeling like [37] simply concatenate 3D volume with 2D feature maps and feed them to a network only using 2D convolutional layers. However, through preliminary experiments, we find this structure performs badly for our task ( §7).…”

Section: Multi-view Hair Modelingmentioning

confidence: 99%

See 2 more Smart Citations

DeepSketchHair: Deep Sketch-Based 3D Hair Modeling

Shen

Zhang

et al. 2021

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

We present DeepSketchHair, a deep learning based tool for interactive modeling of 3D hair from 2D sketches. Given a 3D bust model as reference, our sketching system takes as input a user-drawn sketch (consisting of hair contour and a few strokes indicating the hair growing direction within a hair region), and automatically generates a 3D hair model, which matches the input sketch both globally and locally. The key enablers of our system are two carefully designed neural networks, namely, S2ONet, which converts an input sketch to a dense 2D hair orientation field; and O2VNet, which maps the 2D orientation field to a 3D vector field. Our system also supports hair editing with additional sketches in new views. This is enabled by another deep neural network, V2VNet, which updates the 3D vector field with respect to the new sketches. All the three networks are trained with synthetic data generated from a 3D hairstyle database. We demonstrate the effectiveness and expressiveness of our tool using a variety of hairstyles and also compare our method with prior art.Index Terms-Sketch-based hair modeling, 3D volumetric structure, deep learning, generative adversarial networks !

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Multi-view Hair Modelingmentioning

confidence: 99%

See 1 more Smart Citation

DeepSketchHair: Deep Sketch-Based 3D Hair Modeling

Shen

Zhang

et al. 2021

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

show abstract

“…The CNN is used to predict depth/ normal maps using flow field regression and a confidence map, which gives the ambiguity at each point of the input sketch. As with Delanoy et al (2018), the user first draws a single viewpoint of the object which is rendered as a 3D object. The user can then either further modify the surface by drawing curves over the surface, or providing depth values at sparse sample points, or reuse the frontal sketch to draw the back view of the object.…”

Section: Interactive Sketchesmentioning

confidence: 99%

“…The authors provide a visual comparison between the models generated by their system and those generated by similar user interfaces Delanoy et al (2018) Use a CNN to provide an initial 3D reconstruction which is updated when the user draws another viewpoint. Can be used to model man-made objects with flat, orthogonal faces but through the addition and subtraction of primitives also supports holes and convex protrusions Evaluated by two expert users and six other participants with limited drawing/3D modeling skills.…”

Section: Interactive Sketchesmentioning

confidence: 99%

Sketch-based interaction and modeling: where do we stand?

Bonnici

Akman

Calleja

et al. 2019

AIEDAM

View full text Add to dashboard Cite

Sketching is a natural and intuitive communication tool used for expressing concepts or ideas which are difficult to communicate through text or speech alone. Sketching is therefore used for a variety of purposes, from the expression of ideas on two-dimensional (2D) physical media, to object creation, manipulation, or deformation in three-dimensional (3D) immersive environments. This variety in sketching activities brings about a range of technologies which, while having similar scope, namely that of recording and interpreting the sketch gesture to effect some interaction, adopt different interpretation approaches according to the environment in which the sketch is drawn. In fields such as product design, sketches are drawn at various stages of the design process, and therefore, designers would benefit from sketch interpretation technologies which support these differing interactions. However, research typically focuses on one aspect of sketch interpretation and modeling such that literature on available technologies is fragmented and dispersed. In this paper, we bring together the relevant literature describing technologies which can support the product design industry, namely technologies which support the interpretation of sketches drawn on 2D media, sketch-based search interactions, as well as sketch gestures drawn in 3D media. This paper, therefore, gives a holistic view of the algorithmic support that can be provided in the design process. In so doing, we highlight the research gaps and future research directions required to provide full sketch-based interaction support.

show abstract

Sketch‐based modeling with a differentiable renderer

Xiang

Wang

Jiang

et al. 2020

Computer Animation & Virtual

View full text Add to dashboard Cite

Sketch-based modeling aims to recover three-dimensional (3D) shape from two-dimensional line drawings. However, due to the sparsity and ambiguity of the sketch, it is extremely challenging for computers to interpret line drawings of physical objects. Most conventional systems are restricted to specific scenarios such as recovering for specific shapes, which are not conducive to generalize. Recent progress of deep learning methods have sparked new ideas for solving computer vision and pattern recognition issues. In this work, we present an end-to-end learning framework to predict 3D shape from line drawings. Our approach is based on a two-steps strategy, it converts the sketch image to its normal image, then recover the 3D shape subsequently. A differentiable renderer is proposed and incorporated into this framework, it allows the integration of the rendering pipeline with neural networks. Experimental results show our method outperforms the state-of-art, which demonstrates that our framework is able to cope with the challenges in single sketch-based 3D shape modeling.

show abstract

3D Sketching using Multi-View Deep Volumetric Prediction

Cited by 116 publications

References 76 publications

DeepSketchHair: Deep Sketch-Based 3D Hair Modeling

DeepSketchHair: Deep Sketch-Based 3D Hair Modeling

Sketch-based interaction and modeling: where do we stand?

Sketch‐based modeling with a differentiable renderer

Contact Info

Product

Resources

About