Variational Texture Synthesis with Sparsity and Spectrum Constraints

Abstract

This paper introduces a new approach for texture synthesis. We propose a unified framework that both imposes first order statistical constraints on the use of atoms from an adaptive dictionary, as well as second order constraints on pixel values. This is achieved thanks to a variational approach, the minimization of which yields local extrema, each one being a possible texture synthesis. On the one hand, the adaptive dictionary is created using a sparse image representation rationale, and a global constraint is imposed on the maximal number of use of each atom from this dictionary. On the other hand, a constraint on second order pixel statistics is achieved through the power spectrum of images. In a sense, this method reconciles the first and second theory proposed by B. Julesz for texture discrimination. An advantage of the proposed method is its ability to truly synthesize textures, without verbatim copy of small pieces from the exemplar. In an extensive experimental section, we show that the resulting synthesis achieves state of the art results, both for structured and small scale textures.

Optimal Patch Assignment for Statistically Constrained Texture Synthesis

“…3 shows a comparison of the results obtained using our method with the patch-based approach of Kwatra et al [19], and the statistical matching methods of Portilla et al [22] and Tartavel et al [28]. The two statistical matching methods successfully preserve the color distributions.…”

“…1, that the features from the two images are mostly similar. [19], the two statistical matching methods of Portilla et al [22] and Tartavel et al [28], and the PatchMatch method of Barnes et al [2] (with the implementation of David Tschumperlé for texture synthesis [12]). These approaches are used with the default parameters described in the respective papers.…”

“…For instance, using periodic boundary conditions, any circular shifting of the input image is a global minimizer. Still, as usually done for variational texture synthesis [22,19,28], we minimize the non-convex functional of Problem (1) by alternate minimization with respect to u and σ separately starting from a random image. This relies on the assumption that local minimizers of the functional provide the desired result, which is verified in practice.…”

“…A common way to capture large scale correlations between characteristics in the input is to use a coarse-to-fine synthesis [31,18,28]. We apply this strategy by computing the Gaussian pyramid {u s } S−1 s=0 of the input image, which is composed of S images u s computed by filtering u 0 with a Gaussian kernel with standard deviation 0.8s and sub-sampled with a stepsize of 2 s .…”

“…or empirical distributions. In the literature, a broad range of such characteristics have been investigated such as color histograms, the Fourier modulus [10], steerable filter response histograms [14,6,22,26], and more recently occurrences in an adapted patch dictionary [27,28] or correlation matrices in convolutional neural networks [11]. By essence, these methods are successful in preserving the characteristics of interest.…”