MineGAN: effective knowledge transfer from GANs to target domains with few images

Abstract

One of the attractive characteristics of deep neural networks is their ability to transfer knowledge obtained in one domain to other related domains. As a result, high-quality networks can be trained in domains with relatively little training data. This property has been extensively studied for discriminative networks but has received significantly less attention for generative models. Given the often enormous effort required to train GANs, both computationally as well as in the dataset collection, the re-use of pretrained GANs is a desirable objective. We propose a novel knowledge transfer method for generative models based on mining the knowledge that is most beneficial to a specific target domain, either from a single or multiple pretrained GANs. This is done using a miner network that identifies which part of the generative distribution of each pretrained GAN outputs samples closest to the target domain. Mining effectively steers GAN sampling towards suitable regions of the latent space, which facilitates the posterior finetuning and avoids pathologies of other methods such as mode collapse and lack of flexibility. We perform experiments on several complex datasets using various GAN architectures (BigGAN, Progressive GAN) and show that the proposed method, called MineGAN, effectively transfers knowledge to domains with few target images, outperforming existing methods. In addition, MineGAN can successfully transfer knowledge from multiple pretrained GANs.

Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs

“…• MineGAN [44]: To avoid overfitting of the generator, MineGAN suggests to fix the generator and modify the latent codes. To this end, MineGAN train a miner network that transforms the latent code to another latent code.…”

“…In our experiments, MineGAN totally fails to adapt to the target distribution. Note that MineGAN assumes the source distribution covers (or at least close to) the target distribution (e.g., adult faces to child faces as in the original paper [44]), but cannot be applied if the distributions have disjoint support (e.g., human faces to dog faces).…”

“…Current state-of-the-art GANs, however, often require a large amount of training data and heavy computational resources, which thus limits the applicability of GANs in practical scenarios. Numerous techniques have been proposed to overcome this limitation, e.g., transferring knowledge of a welltrained source model [45,32,44], learning meta-knowledge for quick adaptation to a target domain [24,47,42], using an auxiliary task to facilitate training [7,26,48,49], improving an inference procedure of suboptimal models [2,39,29,38], using an expressive prior distribution [13], actively choosing samples to give supervision for conditional generation [29], or actively sampling mini-batches for training [37].…”

“…Following the success of transferring classifiers in recognition tasks, one can also consider utilizing welltrained GAN backbones for downstream generation tasks. While several methods propose such transfer-learning approaches to training GANs [45,32,44], they are often prone to overfitting with limited training data [45] or not robust in learning a significant distribution shift [32,44].…”

DATGAN: Integrating expert knowledge into deep learning for synthetic tabular data

“…They show that they can add a new class to a pre-trained generator without disturbing the performance of the original domain. Wang et al (2020) propose to use a miner network that identifies which distribution of multiple pre-trained GANs is the most beneficial for a specific target. This mining pushed the sampling towards more suitable regions in the latent space.…”