Wasserstein GANs with Gradient Penalty Compute Congested Transport

Abstract

Wasserstein GANs with Gradient Penalty (WGAN-GP) are an extremely popular method for training generative models to produce high quality synthetic data. While WGAN-GP were initially developed to calculate the Wasserstein 1 distance between generated and real data, recent works (e.g. Stanczuk et al. (2021)) have provided empirical evidence that this does not occur, and have argued that WGAN-GP perform well not in spite of this issue, but because of it. In this paper we show for the first time that WGAN-GP compute the minimum of a different optimal transport problem, the so-called congested transport (Carlier et al. (2008)). Congested transport determines the cost of moving one distribution to another under a transport model that penalizes congestion. For WGAN-GP, we find that the congestion penalty has a spatially varying component determined by the sampling strategy used in Gulrajani et al. ( 2017) which acts like a local speed limit, making congestion cost less in some regions than others. This aspect of the congested transport problem is new in that the congestion penalty turns out to be unbounded and depend on the distributions to be transported, and so we provide the necessary mathematical proofs for this setting. We use our discovery to show that the gradients of solutions to the optimization problem in WGAN-GP determine the time averaged momentum of optimal mass flow. This is in contrast to the gradients of Kantorovich potentials for the Wasserstein 1 distance, which only determine the normalized direction of flow. This may explain, in support of Stanczuk et al. (2021), the success of WGAN-GP, since the training of the generator is based on these gradients.

Trust the Critics: Generatorless and Multipurpose WGANs with Initial Convergence Guarantees

“…In practice we found that this value of λ stabilizes the estimates of W 1 (µ n , ν) and the training of TTC; using smaller values of λ leads to inflated estimates of W 1 (µ n , ν), leading to overly large step sizes and unstable training. This is confirmed by the recent analysis in [30], which shows that at best the value in (15), in expectation, converges to W 1 (µ n , ν) like O(λ −1 ).…”

“…To start with, our theoretical analysis relies on being able to compute a Kantorovich potential u 0 for the pair (µ, ν). Given recent results in [30] showing that the optimization problem for learning critics from [17] actually returns a function that solves a congested transport problem, which is distinct from a Kantorovich potential, one may ask to what extent our theoretical analysis applies to the types of critics learned in practice. The distinction between these problems decreases as λ gets large, so we suspect that our assumption of being able to compute a Kantorovich potential is reasonable for the λ value we used (λ = 1000), but this is certainly something to consider, especially for the smaller values of λ typically used in the literature.…”