Designing an encoder for StyleGAN image manipulation

Tov, Omer; Alaluf, Yuval; Nitzan, Yotam; Patashnik, Or; Cohen–Or, Daniel

doi:10.1145/3476576.3476706

Cited by 117 publications

(189 citation statements)

References 14 publications

Supporting

Mentioning

188

Contrasting

Order By: Relevance

“…Recently, Richardson et al [62] introduced a residual feature pyramid network as the encoder of their pixel2style2pixel framework to extract the extended latent code from a given image efficiently. Many other methods [61], [63], [64] perform the image modification by inverting an image to the latent space.…”

Section: Image Embeddingmentioning

confidence: 99%

See 1 more Smart Citation

Do Inpainting Yourself: Generative Facial Inpainting Guided by Exemplars

Lu¹,

Zhao²,

Jiang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Facial image inpainting is a task of filling visually realistic and semantically meaningful contents for missing or masked pixels in a face image. Although existing methods have made significant progress in achieving high visual quality, the controllable diversity of facial image inpainting remains an open problem in this field. This paper introduces EXE-GAN, a novel diverse and interactive facial inpainting framework, which can not only preserve the high-quality visual effect of the whole image but also complete the face image with exemplar-like facial attributes. The proposed facial inpainting is achieved based on generative adversarial networks by leveraging the global style of input image, the stochastic style, and the exemplar style of exemplar image. A novel attribute similarity metric is introduced to encourage networks to learn the style of facial attributes from the exemplar in a self-supervised way. To guarantee the natural transition across the boundary of inpainted regions, a novel spatial variant gradient backpropagation technique is designed to adjust the loss gradients based on the spatial location. A variety of experimental results and comparisons on public CelebA-HQ and FFHQ datasets are presented to demonstrate the superiority of the proposed method in terms of both the quality and diversity in facial inpainting.

show abstract

Section: Image Embeddingmentioning

confidence: 99%

“…2) Style encoder: A pre-trained pixel2style2pixel style encoder E [62], [63] directly maps an image to a disentangled style latent space W +. Given the pre-trained network parameters θe , the style encoder is used to obtain the exemplar style code…”

Section: A Overviewmentioning

confidence: 99%

Do Inpainting Yourself: Generative Facial Inpainting Guided by Exemplars

Lu¹,

Zhao²,

Jiang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To manipulate real images, it is first necessary to invert them into their latent code representations. This is typically done via per-image optimization [63,38,8,12,1,2,28,54,48] or by training an encoder to learn a direct mapping from a given image to its corresponding latent code [63,45,62,46,47,55,5,9]. For a comprehensive survey on GAN inversion, we refer the reader to Xia et al [60].…”

Section: Real Image Editingmentioning

confidence: 99%

“…For a comprehensive survey on GAN inversion, we refer the reader to Xia et al [60]. In this work, we adopt the e4e encoder from Tov et al [55] as it is designed to facilitate improved editing on the resulting inversions. Many methods resort to manipulating images by traversing the latent space.…”

Section: Real Image Editingmentioning

confidence: 99%

StyleFusion: A Generative Model for Disentangling Spatial Segments

Kafri¹,

Patashnik²,

Alaluf³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present StyleFusion, a new mapping architecture for StyleGAN, which takes as input a number of latent codes and fuses them into a single style code. Inserting the resulting style code into a pre-trained StyleGAN generator results in a single harmonized image in which each semantic region is controlled by one of the input latent codes. Effectively, StyleFusion yields a disentangled representation of the image, providing fine-grained control over each region of the generated image. Moreover, to help facilitate global control over the generated image, a special input latent code is incorporated into the fused representation. StyleFusion operates in a hierarchical manner, where each level is tasked with learning to disentangle a pair of image regions (e.g., the car body and wheels). The resulting learned disentanglement allows one to modify both local, fine-grained semantics (e.g., facial features) as well as more global features (e.g., pose and background), providing improved flexibility in the synthesis process. As a natural extension, Style-Fusion enables one to perform semantically-aware crossimage mixing of regions that are not necessarily aligned. Finally, we demonstrate how StyleFusion can be paired with existing editing techniques to more faithfully constrain the edit to the user's region of interest. Code is available at: https://github.com/OmerKafri/StyleFusion.

show abstract

“…• Data can be mapped to a latent space without the need to train an additional encoder compared to GANs Tov et al [2021], which makes the interpolation between two given input data readily available with only one model.…”

Section: Introduction and Related Workmentioning

confidence: 99%

CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis

Rouard,

Hadjeres

2021

Preprint

View full text Add to dashboard Cite

In this paper, we propose a novel score-base generative model for unconditional raw audio synthesis. Our proposal builds upon the latest developments on diffusion process modeling with stochastic differential equations, which already demonstrated promising results on image generation. We motivate novel heuristics for the choice of the diffusion processes better suited for audio generation, and consider the use of a conditional U-Net to approximate the score function. While previous approaches on diffusion models on audio were mainly designed as speech vocoders in medium resolution, our method termed CRASH (Controllable Raw Audio Synthesis with High-resolution) allows us to generate short percussive sounds in 44.1kHz in a controllable way. Through extensive experiments, we showcase on a drum sound generation task the numerous sampling schemes offered by our method (unconditional generation, deterministic generation, inpainting, interpolation, variations, class-conditional sampling) and propose the class-mixing sampling, a novel way to generate "hybrid" sounds. Our proposed method closes the gap with GAN-based methods on raw audio, while offering more flexible generation capabilities with lighter and easier-to-train models.

show abstract

Designing an encoder for StyleGAN image manipulation

Cited by 117 publications

References 14 publications

Do Inpainting Yourself: Generative Facial Inpainting Guided by Exemplars

Do Inpainting Yourself: Generative Facial Inpainting Guided by Exemplars

StyleFusion: A Generative Model for Disentangling Spatial Segments

CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis

Contact Info

Product

Resources

About