An error analysis of generative adversarial networks for learning distributions

Abstract

This paper studies how well generative adversarial networks (GANs) learn probability distributions from finite samples. Our main results estimate the convergence rates of GANs under a collection of integral probability metrics defined through Hölder classes, including the Wasserstein distance as a special case. We also show that GANs are able to adaptively learn data distributions with low-dimensional structure or have Hölder densities, when the network architectures are chosen properly. In particular, for distributions concentrate around a low-dimensional set, it is proved that the learning rates of GANs do not depend on the high ambient dimension, but on the lower intrinsic dimension. Our analysis is based on a new oracle inequality decomposing the estimation error into generator and discriminator approximation error and statistical error, which may be of independent interest.

Approximation bounds for norm constrained neural networks with applications to regression and GANs

“…Suppose g n,m ∈ G is a solution with optimization error ǫ opt . Using the argument in Huang et al (2021), one can show that g n,m achieves the same rate as g n in Theorem 4.6, if m is sufficiently large.…”

“…Lemma 4.5 (Huang et al (2021)). Assume that F is symmetric (f ∈ F implies −f ∈ F), µ and g # ν are supported on [0, 1] d for all g ∈ G. Then, for any g n ∈ G satisfying (4.8),…”

“…It has been showed that GANs perform extremely well in practice (Gulrajani et al, 2017;Miyato et al, 2018;Brock et al, 2019). We can combine the error analysis in Huang et al (2021) with Theorem 3.2 to derive convergence rate for GANs with norm constrained neural networks as discriminator, which gives statistical guarantee on the performance of GANs, see Theorem 4.6 and Corollary 4.11.…”

“…by assumption. Using standard symmetrization argument (similar to step 2 in the proof of Theorem 3.10), the statistical error E Sn [d H α (µ, µ n )] can be bounded by Rademacher complexity, which can be further bounded by Dudley's entropy integral (see Huang et al (2021) for more details):…”

“…The assumption that the generator approximation error is zero can be fulfilled by sufficiently large neural network class G = N N (W 1 , L 1 ). More precisely, it was shown in (Yang et al, 2022;Huang et al, 2021) that if ν is absolutely continuous and n W 2 1 L 1 then inf g∈G W 1 ( µ n , g # ν) = 0 for any samples S n = {X i } n i=1 .…”