Third Time's the Charm? Image and Video Editing with StyleGAN3

Abstract

StyleGAN is arguably one of the most intriguing and well-studied generative models, demonstrating impressive performance in image generation, inversion, and manipulation. In this work, we explore the recent StyleGAN3 architecture, compare it to its predecessor, and investigate its unique advantages, as well as drawbacks. In particular, we demonstrate that while StyleGAN3 can be trained on unaligned data, one can still use aligned data for training, without hindering the ability to generate unaligned imagery. Next, our analysis of the disentanglement of the different latent spaces of StyleGAN3 indicates that the commonly used W/W+ spaces are more entangled than their StyleGAN2 counterparts, underscoring the benefits of using the StyleSpace for fine-grained editing. Considering image inversion, we observe that existing encoder-based techniques struggle when trained on unaligned data. We therefore propose an encoding scheme trained solely on aligned data, yet can still invert unaligned images. Finally, we introduce a novel video inversion and editing workflow that leverages the capabilities of a fine-tuned Style-GAN3 generator to reduce texture sticking and expand the field of view of the edited video. Code is available on our project page: https://yuval-alaluf.github.io/ stylegan3-editing/.* Denotes equal contribution.

State-of-the-Art in the Architecture, Methods and Applications of StyleGAN

“…Another challenge can be found in the texture-sticking phenomenon observed in StyleGAN1 and StyleGAN2 [Karras et al 2021], which hinders the realism of generated and manipulated videos. To overcome this, Alaluf et al [2022] combine the PTI [Roich et al 2021] and ReStyle [Alaluf et al 2021b] encoding techniques for encoding and editing videos with the StyleGAN3 [Karras et al 2021] generator. Further leveraging the equivariance of StyleGAN3, they demonstrate the ability to expand the field of view when working on a video with a cropped subject resulting in more uniform video editing.…”

Overparameterization Improves StyleGAN Inversion

“…3, left). However, this change also modifies the latent space after training, which in turn damages editability 4 .…”

“…In particular, the extension to W+ uses a different latent code for each layer. This change is a key component of most inversion pipelines [4,15,41,49,57] as it improves reconstruction quality tremendously. But it comes at a price: because the latent space is altered after training, editability suffers.…”

Hypercomplex Image-to-Image Translation

“…Recently, I2I applications are becoming widespread, including a plethora of diverse tasks such as attribute manipulation [1]- [4], sketch-to-image [5], [6], style transfer [7], [8], semantic synthesis [9], [10], and others [11]- [14]. Among these, generative adversarial networks (GANs) [15] are particularly suitable for this task due to few restrictions on the generator network.…”