Deep multispectral painting reproduction via multi-layer, custom-ink printing

Abstract

We propose a workflow for spectral reproduction of paintings, which captures a painting's spectral color, invariant to illumination, and reproduces it using multi-material 3D printing. We take advantage of the current 3D printers' capabilities of combining highly concentrated inks with a large number of layers, to expand the spectral gamut of a set of inks. We use a data-driven method to both predict the spectrum of a printed ink stack and optimize for the stack layout that best matches a target spectrum. This bidirectional mapping is modeled using a pair of neural networks, which are optimized through a problem-specific multi-objective loss function. Our loss function helps find the best possible ink layout resulting in the balance between spectral reproduction and colorimetric accuracy under a multitude of illuminants. In addition, we introduce a novel spectral vector error diffusion algorithm based on combining color contoning and halftoning, which simultaneously solves the layout discretization and color quantization problems, accurately and efficiently. Our workflow outperforms the state-of-the-art models for spectral prediction and layout optimization. We demonstrate reproduction of a number of real paintings and historically important pigments using our prototype implementation that uses 10 custom inks with varying spectra and a resin-based 3D printer.

Mixed integer ink selection for spectral reproduction

“…We demonstrate that for obtaining high-accuracy results, more dataintensive prediction models could be later used for final spectral reproduction using selected inks. In particular, we rely on a datadriven method based on neural networks [Shi et al 2018]. We show that the accuracy of this network can be considerably improved by adapting it to the input content without requiring additional data.…”

“…For actual spectral reproduction, we run an accurate, data-driven, neural network prediction model. This network is very similar to the one proposed by Shi et al [2018] for reproducing spectral oil paintings except we use it for predicting multi-layer halftones while Shi et al [2018] predict contone ink stacks. The goal of the neural spectral separation, composed of a forward and an inverse network as a decoder and encoder respectively, is to take a spectral reflectance as 4 As an example, the convex hull of a point set is a coreset for computing the diameter of the set.…”

“…We demonstrate high-fidelity spectral reproduction through spectral separation using a neural network, a library of 43 inks, and a custom printing setup. Furthermore, we make a simple but important observation about the neural spectral separation [Shi et al 2018] where we adapt the neural network to different inputs leading to a considerable accuracy gain.…”

Programmable Filament

“…Although CleanColor [8] helped resolve the issue of mixed colors appearing in dual-nozzle printing, users are typically occupied with tedious calibrations and required maintenance tasks. Alternatively, an advanced additive-manufacturing technique has been investigated, such as inkjet 3D printing with special materials [24,30,38,32]. Post-processing is another alternative, painting in full colors, spraying electro-magnetic [44] or color-changeable paint [12].…”

Towards spatially varying gloss reproduction for 3D printing

“…Color printing is also possible using only transparent materials that are stacked to produce the desired color [Babaei et al 2017]. Recently, such a technique was also proposed for spectral color reproduction [Shi et al 2019]. Besides ink-jet printers, color can be produced using different technologies such as paper lamination, powder-binder, or fused filament fabrication, but the quality does not match that of ink-jet printing.…”