A Convenient Infinite Dimensional Framework for Generative Adversarial Learning

Abstract

In recent years, generative adversarial networks (GANs) have demonstrated impressive experimental results while there are only a few works that foster statistical learning theory for GANs. In this work, we propose an infinite dimensional theoretical framework for generative adversarial learning. Assuming the class of uniformly bounded k-times α-Hölder differentiable (C k,α ) and uniformly positive densities, we show that the Rosenblatt transformation induces an optimal generator, which is realizable in the hypothesis space of C k,α generators. With a consistent definition of the hypothesis space of discriminators, we further show that in our framework the Jensen-Shannon (JS) divergence between the distribution induced by the generator from the adversarial learning procedure and the data generating distribution converges to zero. As our convenient framework avoids modeling errors both for generators and discriminators, by the error decomposition for adversarial learning it suffices that the sampling error vanishes in the large sample limit. To prove this, we endow the hypothesis spaces of generators and discriminators with C k,α ′ -topologies, 0 < α ′ < α, which render the hypothesis spaces to compact topological spaces such that the uniform law of large numbers can be applied. Under sufficiently strict regularity assumptions on the density of the data generating process, we also provide rates of convergence based on concentration and chaining. To this avail, we first prove subgaussian properties of the empirical process indexed by generators and discriminators. Furthermore, as covering numbers of bounded sets in C k,α -Hölder spaces with respect to the L ∞ -norm lead to a convergent metric entropy integral if k is sufficiently large, we obtain a finite constant in Dudley's inequality. This, in combination with McDiarmid's inequality, provides explicit rate estimates for the convergence of the GAN learner to the true probability distribution in JS divergence.

Generative Modeling of Turbulence

“…The feedback of D to φ is transported backwards by backpropagation [58] through the concatenated mapping D • φ in order to train the weights of the neural network φ. At the same time, the universal approximation property of (deep) neural networks guarantees that any mappings φ and D can be represented with a given precision, provided the architecture of the networks is sufficiently wide and deep, see [23,5,56,64,30,76,65,32] for qualitative and quantitative results.…”

“…Apparently, in the limit T → ∞ by ergodicity (3), the first term converges to the first term in (6) whereas the second term converges almost surely by the law of large numbers. Therefore, the generator φ that is learned from the empirical loss function (13) for large T will approximately solve the minimax problem (5), which by (7) relates to the Jensen-Shannon distance between the estimated measure φT * λ and the invariant measure µ of the ergodic system. In particular, we obtain the following: Theorem 1.…”

“…Here we give a sketch of the proof. For a detailed argument in a related situation, see [5]. We introduce the following notation: D φ is the discriminator solving D φ ∈ arg max L(φ, D, ϕ T ) and, likewise, Dφ ∈ arg max L(φ, D, ϕ T ),…”

“…In the sixth line, we used the definition of D φ0 . In the seventh line, we again used (7) and in the final line we applied (9), which is possible under the given assumptions as proven in [5].…”

“…The second term on the right hand side vanishes by the uniform law of large numbers, as the hypothesis spaces H and H D can be endowed with C k,α topologies that are, for α < α, slightly little weaker than the C k,α -topology. Nevertheless, the hypothesis spaces under these topologies are compact, see [5] for the details. Consequently, the expression in the first term vanishes by the standard uniform law of large numbers, see e.g.…”