StyleFusion: Disentangling Spatial Segments in StyleGAN-Generated Images

Abstract

We present StyleFusion , a new mapping architecture for StyleGAN, which takes as input a number of latent codes and fuses them into a single style code. Inserting the resulting style code into a pre-trained StyleGAN generator results in a single harmonized image in which each semantic region is controlled by one of the input latent codes. Effectively, StyleFusion yields a disentangled representation of the image, providing fine-grained control over each region of the generated image. Moreover, to help facilitate global control over the generated image, a special input latent code is incorporated into the fused representation. StyleFusion operates in a hierarchical manner, where each level is tasked with learning to disentangle a pair of image regions (e.g., the car body and wheels). The resulting learned disentanglement allows one to modify both local, fine-grained semantics (e.g., facial features) as well as more global features (e.g., pose and background), providing improved flexibility in the synthesis process. As a natural extension, StyleFusion allows one to perform semantically-aware cross-image mixing of regions that are not necessarily aligned. Finally, we demonstrate how StyleFusion can be paired with existing editing techniques to more faithfully constrain the edit to the user’s region of interest. Code is available at: https://github.com/OmerKafri/StyleFusion.

CoralStyleCLIP: Co-optimized Region and Layer Selection for Image Editing

“…For ameliorating edits in unexpected regions of an image, strategies for blending latent features have been an emerging theme in many recent papers [7,16,17,21]. [7,16,21] interpolate spatial features more explicitly.…”

“…[7,16,21] interpolate spatial features more explicitly. In contrast, StyleFusion [17] realizes similar objectives through blended latent code extracted using a fusion network that combines disjoint semantic attributes from multiple images into a single photorealistic image.…”

State‐of‐the‐Art in the Architecture, Methods and Applications of StyleGAN

“…Similarly, Abdal et al [AQW20] change the spatial activations to allow scribble‐level control for the user (see Section 4 for more details). StyleFusion [KPACO21] propose a new mapping architecture for StyleGAN to better disentangle a target attribute. This results in a learned blending between style codes, resulting in fine‐grained local control of the edited images.…”

“…Alternatively, Tritong et al [TRS21] directly use StyleGAN and f for segmentation by first inverting a real image into latent space. In the context of local editing, Collins et al [CBPS20] and Kafri et al [KPACO21] perform a simple clustering procedure over StyleGAN's internal representations to obtain a semantic segmentation of an input image. This semantic map can then be used to perform local editing over an image, guided by a target reference image.…”

GAN-Based Face Attribute Editing

“…The latent component responsible for synthesizing the target image region is then located via an optimization algorithm, and FAM can be achieved by modifying it according to the corresponding code of an exemplar image. StyleFusion [163] proposes a hierarchical fusion strategy to disentangle latent representations according to the decomposition patterns obtained by EIS. In addition, RIS [164] extends EIS to manipulate more challenging facial attributes which involve large geometric changes (e.g., hairstyle and pose).…”