Stein Latent Optimization for Generative Adversarial Networks

Abstract

Generative adversarial networks (GANs) with clustered latent spaces can perform conditional generation in a completely unsupervised manner. However, the salient attributes of unlabeled data in the real-world are mostly imbalanced. Existing unsupervised conditional GANs cannot properly cluster the attributes in their latent spaces because they assume uniform distributions of the attributes. To address this problem, we theoretically derive Stein latent optimization that provides reparameterizable gradient estimations of the latent distribution parameters assuming a Gaussian mixture prior in a continuous latent space. Structurally, we introduce an encoder network and a novel contrastive loss to help generated data from a single mixture component to represent a single attribute. We confirm that the proposed method, named Stein Latent Optimization for GANs (SLOGAN), successfully learns the balanced or imbalanced attributes and performs unsupervised tasks such as unsupervised conditional generation, unconditional generation, and cluster assignment even in the absence of information of the attributes (e.g. the imbalance ratio). Moreover, we demonstrate that the attributes to be learned can be manipulated using a small amount of probe data.

Unsupervised Learning of Structured Representations via Closed-Loop Transcription

“…Self-Conditioned GAN (Liu et al, 2020) uses clustering of discriminative features as labels to train. SLOGAN (Hwang et al, 2021) proposes a new conditional contrastive loss (U2C) to learn latent distribution of the data. Note that compared to our work, ClusterGAN and SLOGAN introduce an additional encoder that leads to increased computational complexity.…”

“…By sampling from these compact models, we can conditionally regenerate meaningful samples from computed clusters. This is known as unsupervised conditional image generation (Hwang et al, 2021).…”

“…The fact that human visual memory serves both discriminative tasks (for example, detection and recognition) and generative or predictive tasks (for example, via replay) (Keller & Mrsic-Flogel, 2018;Josselyn & Tonegawa, 2020;Ven et al, 2020) indicates that this goal is achievable. Beyond being possible, these properties are also highly practical -successfully completing generative tasks like unsupervised conditional image generation (Hwang et al, 2021), for example, inherently requires that learned features for different classes be both structured for sampling and discriminative for conditioning. On the other hand, the generative property can serve as a natural regularization to avoid representation collapse.…”