Interpreting the Latent Space of GANs via Correlation Analysis for Controllable Concept Manipulation

Abstract

Generative adversarial nets (GANs) have been successfully applied in many fields like image generation, inpainting, super-resolution and drug discovery, etc., by now, the inner process of GANs is far from been understood. To get deeper insight of the intrinsic mechanism of GANs, in this paper, a method for interpreting the latent space of GANs by analyzing the correlation between latent variables and the corresponding semantic contents in generated images is proposed. Unlike previous methods that focus on dissecting models via feature visualization, the emphasis of this work is put on the variables in latent space, i.e. how the latent variables affect the quantitative analysis of generated results. Given a pretrained GAN model with weights fixed, the latent variables are intervened to analyze their effect on the semantic content in generated images. A set of controlling latent variables can be derived for specific content generation, and the controllable semantic content manipulation be achieved. The proposed method is testified on the datasets Fashion-MNIST and UT Zappos50K, experiment results show its effectiveness.

Face attribute translation with multiple feature perceptual reconstruction assisted by style translator

“…Unlike the above methods which only factorizes between different attribute‐related codes, our factorization disentangles a latent code into both attribute‐relevant and attribute‐irrelevant codes for each attribute. It is important to note that a recent endeavor to achieve end‐to‐end training of both a latent space factorization module and an image generator has been explored in Reference 17. However, it's noteworthy that the generator utilized in Reference 17 operates on an auto‐encoder framework, resulting in comparatively constrained image quality when compared to a GAN.…”

“…It is important to note that a recent endeavor to achieve end‐to‐end training of both a latent space factorization module and an image generator has been explored in Reference 17. However, it's noteworthy that the generator utilized in Reference 17 operates on an auto‐encoder framework, resulting in comparatively constrained image quality when compared to a GAN. Additionally, the factorization process in Reference 17 is once again reliant on subspace projection, which consequently confines its applicability to disentangling attributes solely between them, akin to the approach described in References 15 and 16. Identity consistency maintenance : For maintaining identity consistency, most recent models 1,3,7,9,18 construct reconstruction loss or use cyclic consistency loss to avoid the network losing face information during the operation.…”

Visual interpretability of image-based classification models by generative latent space disentanglement applied to in vitro fertilization

“…Counterfactual explanation methods that incorporate disentanglement can be further partitioned to methods that rely on annotated side information of image properties, for example face images with annotated properties such as hair color, mustache, or skin color (He et al 2019, Gabbay et al 2021, Li et al 2020, and to unconditioned methods that do not use any further data annotations beyond the binary classification labels for training the classification model (Lang et al 2021, Higgins et al 2021, Rodríguez et al 2021. DISCOVER benefits from the advantages of both approaches of counterfactual explanations and attribution based methods.…”

Semi-Supervised Variational Autoencoders for Out-of-Distribution Generation

“…To assess its effectiveness in addressing OOD reconstruction, attribute manipulation, and conditional generation, we compare it against a conditional VAE as a baseline. Furthermore, we compare our model with the CsVAE [54], PCVAE, [55] and MSP [56]. CsVAE, like BtVAAE, is based on a conditional VAE model but uses two latent variables to separate the information correlated with the attributes y into a pre-defined subspace.…”