A Texture Synthesis Model Based on Semi-Discrete Optimal Transport in Patch Space

Abstract

Exemplar-based texture synthesis consists in producing new synthetic images which have the same perceptual characteristics than a given texture sample while exhibiting sufficient innovation (to avoid verbatim copy). In this paper, we propose to address this problem with a model obtained as local transformations of Gaussian random fields. The local transformations operate on 3 × 3 patches and are designed to solve a semi-discrete optimal transport problem in order to reimpose the patch distribution of the exemplar texture. The semi-discrete optimal transport problem is solved with a stochastic gradient algorithm, whose convergence speed is evaluated on several practical transport cases. After studying the properties of such transformed Gaussian random fields, we propose a multiscale extension of the model which aims at preserving the patch distribution of the exemplar texture at multiple scales. Experiments demonstrate that this multiscale model is able to synthesize structured textures while keeping several mathematical guarantees, and with low requirements in synthesis time and memory storage. In particular, a single patch optimal transport map is shown to be better than iterated nearest neighbor assignments in terms of statistical guarantees. Besides, once the model is estimated, the resulting synthesis algorithm is fast and highly parallel since it amounts to perform weighted nearest neighbor patch assignments at each scale.

A Fast Multi-layer Approximation to Semi-discrete Optimal Transport

“…This important result has been used in [1,11,9,7] to optimize weights v when distributions µ and ν are defined on R D for D = 2 or D = 3 dimensions, making use of gradient descent or L-BFGS algorithm. But in higher dimension, exact gradient computation is challenging, and thus following [4,3], one may turn to the Average Stochastic Gradient Descent (ASGD) Algorithm 1 to solve it.…”

“…A common way to formulate texture synthesis is to ask for an image which is as random as possible while respecting a certain number of statistical constraints. Thus, texture synthesis can be addressed by exploiting OT distances to compare distributions of linear or nonlinear filter responses [19], or comparing directly patch distributions [5,3]. In this paper, we build on the model proposed in [3] which performs texture synthesis by applying a well-chosen OT map on 3×3 patches in a coarse to fine manner (starting from a white noise at the lowest resolution).…”

“…Thus, texture synthesis can be addressed by exploiting OT distances to compare distributions of linear or nonlinear filter responses [19], or comparing directly patch distributions [5,3]. In this paper, we build on the model proposed in [3] which performs texture synthesis by applying a well-chosen OT map on 3×3 patches in a coarse to fine manner (starting from a white noise at the lowest resolution). One limitation of this work is that the restriction to 3 × 3 patches prevents the proper reproduction of geometric structures at larger scales.…”

“…Following [3] we are particularly interested in the semi-discrete OT case, where one wishes to send an absolutely continuous probability measure µ onto a discrete probability measure ν while minimizing a transportation cost. For the semi-discrete case, as shown in [1,7], the OT map can be obtained as a weighted nearest-neighbor (NN) assignment, which is parameterized by a finitedimensional vector v that solves a concave optimization problem.…”

“…Unfortunately, these exact methods are very difficult to apply in high dimension (or when the point set supporting ν is large) where exact gradients computations are intractable. To cope with that, one can use stochastic optimization techniques that rely on Monte-Carlo estimations of the gradient, as proposed in [4,3]. But such a stochastic method has a very slow convergence in high dimension, especially when the support of target measure has a large cardinal.…”