Understanding Estimation and Generalization Error of Generative Adversarial Networks

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

“…2) Generalization of GANs (reference [64]): This work investigates the estima- The experiments validate our theoretical results.…”

“…To provide a neat version of thesis with closely correlated topics, this thesis does not include all of the author's works. We briefly talk about some representatives of the author's other research works [56,64,59,123,61,60,125,101,116,47,130,129,133] as follows.…”

$α$-GAN: Convergence and Estimation Guarantees

“…We now consider a setting where we have a limited number of training samples 1 S x = {X 1 ,...,X n } and S z = {Z 1 ,...,Z m } from P r and P Z , respectively. Also, the discriminator and generator classes are typically neural networks; these limitations lead to estimation errors in training GANs [6], [25], [26]. While [26] models the interplay between both the discriminator and generator in the estimation error bounds, those developed in [6], [25] do not explicitly capture the role of the generator.…”

“…Also, the discriminator and generator classes are typically neural networks; these limitations lead to estimation errors in training GANs [6], [25], [26]. While [26] models the interplay between both the discriminator and generator in the estimation error bounds, those developed in [6], [25] do not explicitly capture the role of the generator. We adopt the approach in [26]; to this end, we begin with the notion of neural net (nn) distance (first introduced in [24]) as defined for the setup in [26], [27]:…”

“…where, w k is a parameter vector of the output layer; for i ∈ [1 : k − 1] and j ∈ [1 : l], W i and V j are parameter matrices; r i (•) and s j (•) are entry-wise activation functions of layers i and j, i.e., for a ∈ R t , r i (a) = [r i (a 1 ),...,r i (a t )] and s i (a) = [s i (a 1 ),...,s i (a t )]; and σ(•) is the sigmoid function given by σ(p) = 1/(1 + e −p ) (note that σ does not appear in the discriminator in [26,Equation (7)] as the discriminator considered in the neural net distance is not a soft classifier mapping to [0,1]). We assume that each r i (•) and s j (•) are R iand S j -Lipschitz, respectively, and also that they are positive homogeneous, i.e., r i (λp) = λr i (p) and s j (λp) = λs j (p), for any λ ≥ 0 and p ∈ R. Finally, as modelled in [26], [28]- [30], we assume that the Frobenius norms of the parameter matrices are bounded, i.e.,…”