Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models

Abstract

We introduce a new local sparse attention layer that preserves two-dimensional geometry and locality. We show that by just replacing the dense attention layer of SAGAN with our construction, we obtain very significant FID, Inception score and pure visual improvements. FID score is improved from 18.65 to 15.94 on ImageNet, keeping all other parameters the same. The sparse attention patterns that we propose for our new layer are designed using a novel information theoretic criterion that uses information flow graphs.We also present a novel way to invert Generative Adversarial Networks with attention. Our method uses the attention layer of the discriminator to create an innovative loss function. This allows us to visualize the newly introduced attention heads and show that they indeed capture interesting aspects of two-dimensional geometry of real images.

Image Restoration from Parametric Transformations using Generative Models

“…The generator GpZq can be a neural network (deep or shallow), trained with the help of adversarial [4,7,9,18,34] or non-adversarial techniques [8,14]. If a non-adversarial training method is adopted then this clearly suggests that a discriminator function, which plays an important role in the techniques proposed in [13,40,45,46], does not exist. For this reason our goal, similarly to [5,10,48], is to propose estimation techniques that do not rely on discriminator functions.…”

“…, Z L u of hpZq and by evoking the the Law of Large Numbers we approximate the MMSE estimate. For the MAP estimates in (13) we have…”

“…Unlike our proposed technique, existing methods do not perform this combined optimization and are therefore unable to restore the original image when the transformation contains unknown parameters. As in [13,45] we ran our simulations for every testing image three times, with different initializations and retain the solution with the smallest reconstruction error. In Figure 1 we present the corresponding results for the problem of horizontal blurring.…”

“…For inpainting such methods are considered in [40,45,46], with the superresolution and coloring problem also being mentioned in the same articles. The inversion of a generative model enjoys exactly the same formulation and suitable solutions can be found in [13,27]. Finally, for the image separation problem recent efforts based on generative modeling can be found in [3,25,41,43].…”

“…Early efforts based on generative modeling [13,25,27,40,45,46] were using the generative model partially (only the generator function). Only recently [3,5,41,43,48] we see techniques that employ the complete model (generator function and input density) improving the performance of the corresponding methods.…”