ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement

Alaluf, Yuval; Patashnik, Or; Cohen–Or, Daniel

doi:10.1109/iccv48922.2021.00664

Cited by 273 publications

(156 citation statements)

References 24 publications

Supporting

Mentioning

156

Contrasting

Order By: Relevance

“…Wei et al [2021] and Hybrid approach where pSp [Richardson et al 2021] and optimization to W+ are employed, and PTI [Roich et al 2021]. Second raw demonstrates the inversion using encoders, IDInvert [Zhu et al 2020c], pSp [Richardson et al 2021], e4e and ReStyle [Alaluf et al 2021b], over the same input image. The third row illustrates the editability of different regions in the latent space.…”

Section: Gan Inversionmentioning

confidence: 99%

“…While encoder-based techniques result in an efficient inference scheme, taking a fraction of a second per image, the reconstructions are typically less accurate than optimization-based approaches. In an attempt to close the gap between the two methodologies, Alaluf et al [2021b] introduce an iterative refinement scheme over standard encoder-based inversion techniques. Instead of directly outputting the inferred latent code using a single forward pass through the network, the encoder outputs a sequence of residuals used to iteratively improve the inverted latent code and corresponding reconstruction.…”

Section: Gan Inversionmentioning

confidence: 99%

See 1 more Smart Citation

State-of-the-Art in the Architecture, Methods and Applications of StyleGAN

Bermano¹,

Gal²,

Alaluf³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) have established themselves as a prevalent approach to image synthesis. Of these, StyleGAN offers a fascinating case study, owing to its remarkable visual quality and an ability to support a large array of downstream tasks. This state-of-the-art report covers the StyleGAN architecture, and the ways it has been employed since its conception, while also analyzing its severe limitations. It aims to be of use for both newcomers, who wish to get a grasp of the field, and for more experienced readers that might benefit from seeing current research trends and existing tools laid out. Among StyleGAN's most interesting aspects is its learned latent space. Despite being learned with no supervision, it is surprisingly well-behaved and remarkably disentangled. Combined with StyleGAN's visual quality, these properties gave rise to unparalleled editing capabilities. However, the control offered by StyleGAN is inherently limited to the generator's learned distribution, and can only be applied to images generated by StyleGAN itself. Seeking to bring StyleGAN's latent control to real-world scenarios, the study of GAN inversion and latent space embedding has quickly gained in popularity. Meanwhile, this same study has helped shed light on the inner workings and limitations of StyleGAN. We map out StyleGAN's impressive story through these investigations, and discuss the details that have made StyleGAN the go-to generator. We further elaborate on the visual priors StyleGAN constructs, and discuss their use in downstream discriminative tasks. Looking forward, we point out StyleGAN's limitations and speculate on current trends and promising directions for future research, such as task and target specific fine-tuning.

show abstract

Section: Gan Inversionmentioning

confidence: 99%

Section: Gan Inversionmentioning

confidence: 99%

State-of-the-Art in the Architecture, Methods and Applications of StyleGAN

Bermano¹,

Gal²,

Alaluf³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…To manipulate real images, it is first necessary to invert them into their latent code representations. This is typically done via per-image optimization [63,38,8,12,1,2,28,54,48] or by training an encoder to learn a direct mapping from a given image to its corresponding latent code [63,45,62,46,47,55,5,9]. For a comprehensive survey on GAN inversion, we refer the reader to Xia et al [60].…”

Section: Real Image Editingmentioning

confidence: 99%

StyleFusion: A Generative Model for Disentangling Spatial Segments

Kafri¹,

Patashnik²,

Alaluf³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present StyleFusion, a new mapping architecture for StyleGAN, which takes as input a number of latent codes and fuses them into a single style code. Inserting the resulting style code into a pre-trained StyleGAN generator results in a single harmonized image in which each semantic region is controlled by one of the input latent codes. Effectively, StyleFusion yields a disentangled representation of the image, providing fine-grained control over each region of the generated image. Moreover, to help facilitate global control over the generated image, a special input latent code is incorporated into the fused representation. StyleFusion operates in a hierarchical manner, where each level is tasked with learning to disentangle a pair of image regions (e.g., the car body and wheels). The resulting learned disentanglement allows one to modify both local, fine-grained semantics (e.g., facial features) as well as more global features (e.g., pose and background), providing improved flexibility in the synthesis process. As a natural extension, Style-Fusion enables one to perform semantically-aware crossimage mixing of regions that are not necessarily aligned. Finally, we demonstrate how StyleFusion can be paired with existing editing techniques to more faithfully constrain the edit to the user's region of interest. Code is available at: https://github.com/OmerKafri/StyleFusion.

show abstract

“…Hence, editing a real image starts with finding its latent representation. This process, called GAN inversion, has recently drawn considerable attention [1,4,24,32,38,44]. Early attempts inverted the image to W -StyleGAN's native latent space.…”

Section: Introductionmentioning

confidence: 99%

“…Tov et al [38] define this conflict as the distortion-editability tradeoff, and show that the closer the codes are to W, the better their editability is. Indeed, recent works [4,38,46] suggest a compromise between edibility and distortion, by picking latent codes in W+ which are more editable.…”

Section: Introductionmentioning

confidence: 99%

Pivotal Tuning for Latent-based Editing of Real Images

Roich¹,

Mokady²,

Bermano³

et al. 2021

Preprint

Self Cite

125

View full text Add to dashboard Cite

Recently, a surge of advanced facial editing techniques have been proposed that leverage the generative power of a pre-trained StyleGAN. To successfully edit an image this way, one must first project (or invert) the image into the pre-trained generator's domain. As it turns out, however, StyleGAN's latent space induces an inherent tradeoff between distortion and editability, i.e. between maintaining the original appearance and convincingly altering some of its attributes. Practically, this means it is still challenging to apply ID-preserving facial latent-space editing to faces which are out of the generator's domain. In this paper, we present an approach to bridge this gap. Our technique slightly alters the generator, so that an out-of-domain image is faithfully mapped into an in-domain latent code. The key idea is pivotal tuning -a brief training process that preserves the editing quality of an in-domain latent region, while changing its portrayed identity and appearance. In Pivotal Tuning Inversion (PTI), an initial inverted latent code serves as a pivot, around which the generator is finedtuned. At the same time, a regularization term keeps nearby identities intact, to locally contain the effect. This surgical training process ends up altering appearance features that represent mostly identity, without affecting editing capabilities. To supplement this, we further show that pivotal tuning can also adjust the generator to accommodate a multitude of faces, while introducing negligible distortion on the rest of the domain. We validate our technique through inversion and editing metrics, and show preferable scores to state-of-the-art methods. We further qualitatively demonstrate our technique by applying advanced edits (such as pose, age, or expression) to numerous images of well-known and recognizable identities. Finally, we demonstrate resilience to harder cases, including heavy make-up, elaborate hairstyles and/or headwear, which otherwise could not have been successfully inverted and edited by state-of-the-art methods.

show abstract

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement

Cited by 273 publications

References 24 publications

State-of-the-Art in the Architecture, Methods and Applications of StyleGAN

State-of-the-Art in the Architecture, Methods and Applications of StyleGAN

StyleFusion: A Generative Model for Disentangling Spatial Segments

Pivotal Tuning for Latent-based Editing of Real Images

Contact Info

Product

Resources

About