Differentiable vector graphics rasterization for editing and learning

Li, Tzu‐Mao; Lukáč, Michal; Gharbi, Michaël; Ragan-Kelley, Jonathan

doi:10.1145/3414685.3417871

Cited by 99 publications

(91 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work has developed different techniques to differentiate the rendering computation. The derivation above is similar to Li et al [2018a], and is closely related to the Reynolds transport theorem Li et al 2020b;Zhang et al 2020Zhang et al , 2019. An alternative approach is to apply reparametrizations to remove the discontinuities [Loubet et al 2019], which turns out to be equivalent to applying divergence theorem on the derivative line integrals [Bangaru et al 2020].…”

Section: Case Study I: 2d Differentiable Renderingmentioning

confidence: 99%

“…Some methods ignore geometry derivatives [Gkioulekas et al 2013;Nimier-David et al 2019], some approximate the Dirac delta contributions [de La Gorce et al 2011;Kato et al 2018;Loper and Black 2014], some smooth out the discontinuities [Liu et al 2019], and some apply smooth postprocessing using the geometry buffer [Laine et al 2020]. Meanwhile, other methods derive the correct derivatives using Dirac deltas [Li et al 2018a], reparametrization [Loubet et al 2019], or Reynolds transport theorem [Bangaru et al 2020;Li et al 2020b;Roger et al 2005;Zhang et al 2020Zhang et al , 2019. Differentiable renderers have also been used for inverse shader designs [Guo et al 2020;Shi et al 2020].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Systematically differentiating parametric discontinuities

Bangaru¹,

Michel²,

Mu³

et al. 2021

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

Emerging research in computer graphics, inverse problems, and machine learning requires us to differentiate and optimize parametric discontinuities. These discontinuities appear in object boundaries, occlusion, contact, and sudden change over time. In many domains, such as rendering and physics simulation, we differentiate the parameters of models that are expressed as integrals over discontinuous functions. Ignoring the discontinuities during differentiation often has a significant impact on the optimization process. Previous approaches either apply specialized hand-derived solutions, smooth out the discontinuities, or rely on incorrect automatic differentiation. We propose a systematic approach to differentiating integrals with discontinuous integrands, by developing a new differentiable programming language. We introduce integration as a language primitive and account for the Dirac delta contribution from differentiating parametric discontinuities in the integrand. We formally define the language semantics and prove the correctness and closure under the differentiation, allowing the generation of gradients and higher-order derivatives. We also build a system, Teg, implementing these semantics. Our approach is widely applicable to a variety of tasks, including image stylization, fitting shader parameters, trajectory optimization, and optimizing physical designs.

show abstract

Section: Case Study I: 2d Differentiable Renderingmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Systematically differentiating parametric discontinuities

Bangaru¹,

Michel²,

Mu³

et al. 2021

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recent works have explored using deep learning for vectorization [Carlier et al 2020;Das et al 2021;Egiazarian et al 2020;Gao et al 2019;Guo et al 2019;Kim et al 2018;Li et al 2020;Liu et al 2017;Lopes et al 2019;Reddy et al 2021;Zhou et al 2019a]. These approaches either cast vectorization as a segmentation task [Kim et al 2018], or predict a fixed number of curves either via direct vector supervision [Carlier et al 2020;Lopes et al 2019] or via a differentiable rasterizer [Li et al 2020]. Other approaches improve on that by designing RNNs to predict splines from images [Gao et al 2019;Reddy et al 2021].…”

Section: Input Bitmapmentioning

confidence: 99%

Keypoint-driven line drawing vectorization via PolyVector flow

et al. 2021

View full text Add to dashboard Cite

Line drawing vectorization is a daily task in graphic design, computer animation, and engineering, necessary to convert raster images to a set of curves for editing and geometry processing. Despite recent progress in the area, automatic vectorization tools often produce spurious branches or incorrect connectivity around curve junctions; or smooth out sharp corners. These issues detract from the use of such vectorization tools, both from an aesthetic viewpoint and for feasibility of downstream applications (e.g., automatic coloring or inbetweening). We address these problems by introducing a novel line drawing vectorization algorithm that splits the task into three components: (1) finding keypoints, i.e., curve endpoints, junctions, and sharp corners; (2) extracting drawing topology, i.e., finding connections between keypoints; and (3) computing the geometry of those connections. We compute the optimal geometry of the connecting curves via a novel geometric flow --- PolyVector Flow --- that aligns the curves to the drawing, disambiguating directions around Y-, X-, and T-junctions. We show that our system robustly infers both the geometry and topology of detailed complex drawings. We validate our system both quantitatively and qualitatively, demonstrating that our method visually outperforms previous work.

show abstract

“…They firstly transform the predicted vector parameters into raster drawing images, and then optimize the model at raster level. The transformation is achieved by using an external black-box rendering simulator [Ganin et al 2018;Mellor et al 2019], or a differentiable rendering module Li et al 2020;Nakano 2019;Zheng et al 2019]. Among works that do not require training vector data, Learning-To-Paint is the approach closest to our framework, albeit with some noticeable differences.…”

Section: Related Work 21 Vector Graphics Generationmentioning

confidence: 99%

General virtual sketching framework for vector line art

Simo-Serra

Gao

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

Vector line art plays an important role in graphic design, however, it is tedious to manually create. We introduce a general framework to produce line drawings from a wide variety of images, by learning a mapping from raster image space to vector image space. Our approach is based on a recurrent neural network that draws the lines one by one. A differentiable rasterization module allows for training with only supervised raster data. We use a dynamic window around a virtual pen while drawing lines, implemented with a proposed aligned cropping and differentiable pasting modules. Furthermore, we develop a stroke regularization loss that encourages the model to use fewer and longer strokes to simplify the resulting vector image. Ablation studies and comparisons with existing methods corroborate the efficiency of our approach which is able to generate visually better results in less computation time, while generalizing better to a diversity of images and applications.

show abstract

Differentiable vector graphics rasterization for editing and learning

Cited by 99 publications

References 50 publications

Systematically differentiating parametric discontinuities

Systematically differentiating parametric discontinuities

Keypoint-driven line drawing vectorization via PolyVector flow

General virtual sketching framework for vector line art

Contact Info

Product

Resources

About