AgileGAN

Abstract

Portraiture as an art form has evolved from realistic depiction into a plethora of creative styles. While substantial progress has been made in automated stylization, generating high quality stylistic portraits is still a challenge, and even the recent popular Toonify suffers from several artifacts when used on real input images. Such StyleGAN-based methods have focused on finding the best latent inversion mapping for reconstructing input images; however, our key insight is that this does not lead to good generalization to different portrait styles. Hence we propose AgileGAN, a framework that can generate high quality stylistic portraits via inversion-consistent transfer learning. We introduce a novel hierarchical variational autoencoder to ensure the inverse mapped distribution conforms to the original latent Gaussian distribution, while augmenting the original space to a multi-resolution latent space so as to better encode different levels of detail. To better capture attribute-dependent stylization of facial features, we also present an attribute-aware generator and adopt an early stopping strategy to avoid overfitting small training datasets. Our approach provides greater agility in creating high quality and high resolution (1024×1024) portrait stylization models, requiring only a limited number of style exemplars (~100) and short training time (~1 hour). We collected several style datasets for evaluation including 3D cartoons, comics, oil paintings and celebrities. We show that we can achieve superior portrait stylization quality to previous state-of-the-art methods, with comparisons done qualitatively, quantitatively and through a perceptual user study. We also demonstrate two applications of our method, image editing and motion retargeting.

College students’ perceptions of AI-based writing learning tools: With a focus on Google Translate, Naver Papago, and Grammarly

“…We compare our results with those from the following related works. CycleGAN [21] and U-GAT-IT [7] are image-to-image translation-based methods, and Toonify [13] and AgileGAN [17] are StyleGAN-based portrait stylization approaches. Image-to-image translation frameworks are trained with VoxCeleb2 and style images dataset.…”

“…Especially, StyleGAN [6], which is a generative adversarial network (GAN) based on style latents, has greatly improved the performance of human face generation and extended the range of applications as it allows to modify the style of the generated images with a simple modification of style vectors which modulate the generator. To name a few, [3,17,19] have shown impressive results in image-level translation, in which they utilize GAN inversion techniques [1,8,14] to map the source image into latent space and decode it with a StyleGAN to generate a stylized image. However, despite their impressive performances in image-level portrait stylization, they show limited capability in the video-level translation as can be seen in Fig.…”

State‐of‐the‐Art in the Architecture, Methods and Applications of StyleGAN

“…Such areas further push the latent code outside the distribution over which the generator was trained on. As the distribution of 𝒲 cannot be explicitly modeled, keeping the latent code in the trained distribution is a challenging task. To alleviate the need of preserving the latent code inside the distribution of W , it is possible to work with an extension of Z instead of W. Similarly to the definition of 𝒲+, in 𝒵+ [SLL*21] a different latent code is sampled for each layer of the synthesis network (e.g., 18 for a 1024 × 1024‐resolution generator). Note, that in S there is no notion of 𝒮+ as the latent codes for each layer are different by design.…”

“…Several works make use of a similar form of fine‐tuning. Song et al [SLL*21] introduce several synthesis paths within the generator for different attributes, achieving high‐quality portrait stylization. Following the fine‐tuning process, Jang et al [JJJ*21] leverage the semantic correspondence between the two models.…”

Wildfire Smoke Classification Based on Synthetic Images and Pixel- and Feature-Level Domain Adaptation

“…In order to increase the diversity of a dataset, the current mainstream approaches can be broadly divided into two categories, one based on GANs [ 6 ] to generate the data and the other using synthetic images to overcome the shortage of wildfire smoke data. GANs (generative adversarial networks) [ 6 ] have been used for data enhancement of wildfire smoke, which are trained with the idea of adversarial training and have achieved remarkable results in face generation and many other areas [ 7 , 8 , 9 , 10 ]. Namozov et al [ 11 ] used a GAN network to generate fire images with winter and night backgrounds by the original images and added different seasons and times of the day.…”