E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion

Wei, Tianyi; Chen, Dongdong; Zhou, Wenbo; Liao, Jing; Zhang, Weiming; Liu, Yuan; Hua, Gang; Yu, Nenghai

doi:10.1109/tip.2022.3167305

Cited by 50 publications

(18 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are usually two ways to restore latent codes. Researches such as PULSE and Image2stylegan [23], [24] use the method of optimizing loss, and methods such as pSp [25], e4e [26], E2Style [27] train an encoder through a large number of samples.…”

Section: Use Of Pre-trained Gansmentioning

confidence: 99%

A Unified Framework From Face Image Restoration to Data Augmentation Using Generative Prior

et al. 2023

View full text Add to dashboard Cite

In the domain of data enhancement, image restoration and data augmentation are two tasks gaining increasing attention. Current image restoration models focus on improving clarity using pre-trained generative models, and data augmentation methods try to generate new samples with the help of generative models. These two related topics have long been studied completely separately. We propose a downstream-friendly restoration framework based on pre-trained generative models with the capability of data augmentation for face images. We carefully design our framework to achieve high fidelity when inheriting the generation ability from the pre-trained generator. To achieve this goal, we use a modified U-Net to predict the biases of latent codes and feature maps to guide the generator. We further propose to adopt linear interpolation as an approach to enriching the datasets for downstream tasks, especially for classimbalanced tasks. Effectiveness of our method is demonstrated through experiments on three datasets and one downstream task.

show abstract

Section: Use Of Pre-trained Gansmentioning

confidence: 99%

A Unified Framework From Face Image Restoration to Data Augmentation Using Generative Prior

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Wei et al [WCZ*21] and Alaluf et al [APCO21b] find that a complex hierarchical encoder is unnecessary, especially in unstructured domains (e.g., cars, churches, horses) and instead propose simpler backbones. Wei et al [WCZ*21] further replace the 18 map2style blocks with a simple block comprised of a single average pooling layer and fully connected layer. Rather than encoding an image into a set of style vectors , Kim et al [KCK*21] instead invert images into an intermediate latent space with a spatial dimension, resulting in more accurate reconstructions compared to other encoder networks.…”

Section: Encoding and Inversionmentioning

confidence: 99%

“…Tov et al [TAN*21] extend this to additional domains by employing a similarity loss based on a MoCo [CFGH20] feature extractor pre‐trained on ImageNet. Wei et al [WCZ*21] utilize a pre‐trained face parsing network to achieve more localized supervision during the encoder training.…”

Section: Encoding and Inversionmentioning

confidence: 99%

“…Other works [WCZ*21, ZLW*21, YQQ*21] examined the task of face swapping by blending the latent representations of two input images embedded via a learned encoder. Zhu et al [ZAFW21] study the task of hairstyle transfer.…”

Section: Encoding and Inversionmentioning

confidence: 99%

“…For extracting the learned styles from the encoder's feature maps, Richardson et al [RAP*21] introduce 18 separate map2style modules, one for each input layer of Style‐GAN. Wei et al [WCZ*21] and Alaluf et al [APCO21b] find that a complex hierarchical encoder is unnecessary, especially in unstructured domains (e.g., cars, churches, horses) and instead propose simpler backbones. Wei et al [WCZ*21] further replace the 18 map2style blocks with a simple block comprised of a single average pooling layer and fully connected layer.…”

Section: Encoding and Inversionmentioning

confidence: 99%

See 2 more Smart Citations

State‐of‐the‐Art in the Architecture, Methods and Applications of StyleGAN

Bermano

Gal

Alaluf

et al. 2022

Computer Graphics Forum

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) have established themselves as a prevalent approach to image synthesis. Of these, StyleGAN offers a fascinating case study, owing to its remarkable visual quality and an ability to support a large array of downstream tasks. This state-of-the-art report covers the StyleGAN architecture, and the ways it has been employed since its conception, while also analyzing its severe limitations. It aims to be of use for both newcomers, who wish to get a grasp of the field, and for more experienced readers that might benefit from seeing current research trends and existing tools laid out. Among StyleGAN's most interesting aspects is its learned latent space. Despite being learned with no supervision, it is surprisingly well-behaved and remarkably disentangled. Combined with StyleGAN's visual quality, these properties gave rise to unparalleled editing capabilities. However, the control offered by StyleGAN is inherently limited to the generator's learned distribution, and can only be applied to images generated by StyleGAN itself. Seeking to bring StyleGAN's latent control to real-world scenarios, the study of GAN inversion and latent space embedding has quickly gained in popularity. Meanwhile, this same study has helped shed light on the inner workings and limitations of StyleGAN. We map out StyleGAN's impressive story through these investigations, and discuss the details that have made StyleGAN the go-to generator. We further elaborate on the visual priors StyleGAN constructs, and discuss their use in downstream discriminative tasks. Looking forward, we point out StyleGAN's limitations and speculate on current trends and promising directions for future research, such as task and target specific fine-tuning.

show abstract