Texture Synthesis with Spatial Generative Adversarial Networks

Jetchev, Nikolay; Bergmann, Urs; Vollgraf, Roland

doi:10.48550/arxiv.1611.08207

Cited by 54 publications

(93 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Generative Adversarial Networks (GANs), which have shown remarkable capabilities in image generation in multiple domains [49], [50], [51], can learn those features from data. Specifically, in texture synthesis, they have proven successful at generating new samples of textures from a single input image [12] or from a dataset of images [14], [15], [16], [17], [18]. We build upon the method of Zhou et al [12] which shows good performance on the synthesis of non-stationary single image textures, and extend it to synthesize texture stacks, as well as generate tileable outputs.…”

Section: Texture Synthesismentioning

confidence: 99%

“…For this reason, such a process is frequently done manually by artists in 3D digitization pipelines. Recent advances in Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) have been applied to texture synthesis problems [12], [14], [15], [16], [17], [18], [19] showing unprecedented levels of realism and quality, however, the output of these methods is not tileable. Despite recent methods [13], [16], [20], [21], [22], [23] addressing the problem of tileable texture synthesis, we show that they either assume a particular level of regularity or the generated textures lose a significant amount of visual fidelity with respect to the input exemplars.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps

Rodriguez-Pardo,

Garces

2022

Preprint

View full text Add to dashboard Cite

Real-time graphics applications require high-quality textured materials to convey realism in virtual environments. Generating these textures is challenging as they need to be visually realistic, seamlessly tileable, and have a small impact on the memory consumption of the application. For this reason, they are often created manually by skilled artists. In this work, we present SeamlessGAN, a method capable of automatically generating tileable texture maps from a single input exemplar. In contrast to most existing methods, focused solely on solving the synthesis problem, our work tackles both problems, synthesis and tileability, simultaneously. Our key idea is to realize that tiling a latent space within a generative network trained using adversarial expansion techniques produces outputs with continuity at the seam intersection that can then be turned into tileable images by cropping the central area. Since not every value of the latent space is valid to produce high-quality outputs, we leverage the discriminator as a perceptual error metric capable of identifying artifact-free textures during a sampling process. Further, in contrast to previous work on deep texture synthesis, our model is designed and optimized to work with multi-layered texture representations, enabling textures composed of multiple maps such as albedo, normals, etc. We extensively test our design choices for the network architecture, loss function, and sampling parameters. We show qualitatively and quantitatively that our approach outperforms previous methods and works for textures of different types.

show abstract

Section: Texture Synthesismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps

Rodriguez-Pardo,

Garces

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Infinite image generation. Existing works on infinite image generation mainly consider the generation of only texture-like and pattern-like images [27,5,14,39,55], making it similar to procedural generation [50,47]. SinGAN [56] learn a GAN model from a single image and is able to produce its (potentially unbounded) variations.…”

Section: Related Workmentioning

confidence: 99%

“…Most of the prior work on infinite image generation focused on the synthesis of homogeneous texture-like patterns [27,5,39] and did not explore the infinite generation of complex scenes, like nature or city landscapes. The critical challenge of generating such images compared to texture synthesis is making the produced frames globally consistent with one another: when a scene spans across several frames, they should all be conditioned on some shared information.…”

Section: Introductionmentioning

confidence: 99%

Aligning Latent and Image Spaces to Connect the Unconnectable

Skorokhodov¹,

Sotnikov²,

Elhoseiny³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Learning Unconditional GANs from a Single Image. There are several work on learning GANs from a single texture image [2,22,34]. Recently, a SinGAN approach [46] has shown surprisingly good results on learning unconditional GANs from a single non-texture image.…”

Section: Related Work and Our Contributionsmentioning

confidence: 99%

Learning Inception Attention for Image Synthesis and Image Recognition

Shen¹,

Wu²

2021

Preprint

View full text Add to dashboard Cite

Image synthesis and image recognition have witnessed remarkable progress, but often at the expense of computationally expensive training and inference. Learning lightweight yet expressive deep model has emerged as an important and interesting direction. Inspired by the well-known split-transform-aggregate design heuristic in the Inception building block, this paper proposes a Skip-Layer Inception Module (SLIM) that facilitates efficient learning of image synthesis models, and a same-layer variant (dubbed as SLIM too) as a stronger alternative to the well-known ResNeXts for image recognition. The proposed method is built on the idea of developing spatially-adaptive feature modulation/attention in a network. Specifically, SLIM is done from an input source feature map to a target one, either skip-layer or same-layer. In SLIM, the input feature map is first split into a number of groups (e.g., 4). Each group is then transformed to a latent style vector (via channel-wise attention) and a latent spatial mask (via spatial attention). The learned latent masks and latent style vectors are aggregated to modulate the target feature map. For generative learning, SLIM is built on a recently proposed lightweight Generative Adversarial Networks (i.e., FastGANs) which present a skip-layer excitation (SLE) module. For few-shot image synthesis tasks, the proposed SLIM achieves better performance than the SLE work and other related methods. For one-shot image synthesis tasks, it shows stronger capability of preserving image structures than prior arts such as the SinGANs. For image classification tasks, the proposed SLIM is used as a drop-in replacement for convolution layers in ResNets (resulting the ResNeXt-like models) and achieves better accuracy in the ImageNet-1000 dataset, with significantly smaller model complexity.

show abstract

Texture Synthesis with Spatial Generative Adversarial Networks

Cited by 54 publications

References 7 publications

SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps

SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps

Aligning Latent and Image Spaces to Connect the Unconnectable

Learning Inception Attention for Image Synthesis and Image Recognition

Contact Info

Product

Resources

About