MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking

Abstract

In the past few years, Generative Adversarial Networks (GANs) have dramatically advanced our ability to represent and parameterize high-dimensional, non-linear image manifolds. As a result, they have been widely adopted across a variety of applications, ranging from challenging inverse problems like image completion, to problems such as anomaly detection and adversarial defense. A recurring theme in many of these applications is the notion of projecting an image observation onto the manifold that is inferred by the generator. In this context, Projected Gradient Descent (PGD) has been the most popular approach, which essentially optimizes for a latent vector that minimizes the discrepancy between a generated image and the given observation. However, PGD is a brittle optimization technique that fails to identify the right projection (or latent vector) when the observation is corrupted, or perturbed even by a small amount. Such corruptions are common in the real world, for example images in the wild come with unknown crops, rotations, missing pixels, or other kinds of non-linear distributional shifts which break current encoding methods, rendering downstream applications unusable. To address this, we propose corruption mimicking -a new robust projection technique, that utilizes a surrogate network to approximate the unknown corruption directly at test time, without the need for additional supervision or data augmentation. The proposed method is significantly more robust than PGD and other competing methods under a wide variety of corruptions, thereby enabling a more effective use of GANs in real-world applications. More importantly, we show that our approach produces state-of-the-art performance in several GAN-based applications -anomaly detection, domain adaptation, and adversarial defense, that benefit from an accurate projection.

Anomalous Example Detection in Deep Learning: A Survey

“…Approaches such as [68] and [69] rely on projecting the test image to benign dataset manifold to detect adversarial examples. The underlying assumption in these approaches is that adversarial perturbations move the test image away from the benign image manifold and the effect of adversary can be nullified by projecting the images back onto the benign manifold before classifying them.…”

“…For example, [68] use a sample approximation comprising a database of billions of natural images. On the other hand, [69] use a generative model trained on benign images to estimate the manifold. Given the estimated benign manifold, the projection is done by nearest neighbor search in [68] and gradient-based search in [69].…”

Generative adversarial networks unlock new methods for cognitive science

“…If successful, feeding the resulting latent vector to the generator should faithfully reconstruct the original image. While this is far from a trivial task [22][23][24], substantial progress has been made [19,22,[25][26][27]. In one of the most recent contributions [22], the authors demonstrated how their inversion method enables modifying facial attributes, interpolation, and diffusing the inner parts of one face image into the outer parts of another (Figure 4D).…”

Using Deep Image Priors to Generate Counterfactual Explanations

“…When a pre-trained generative model is available, one can always reproject an OOD sample onto the image manifold, e.g., MimicGAN [2], to produce meaningful reconstructions. However, in this paper, we consider the setup where we assume no access to the original data (or a generative model), but only to a black-box predictive model.…”