Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models

Huang, Chin-Wei; Dinh, Laurent; Courville, Aaron

doi:10.48550/arxiv.2002.07101

Cited by 23 publications

(43 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…VAEs VDVAE (Child, 2020) 2.87 -NVAE (Vahdat & Kautz, 2020) 2.91 23.49 BIVA (Maaløe et al, 2019) 3.08 -Flows FFJORD (Grathwohl et al, 2018) 3.40 -VFlow (Chen et al, 2020) 2.98 -ANF (Huang et al, 2020) 3.05 -…”

Section: Resultsmentioning

confidence: 99%

Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory

Chen¹,

Liu²,

Theodorou³

2021

Preprint

View full text Add to dashboard Cite

Schrödinger Bridge (SB) is an optimal transport problem that has received increasing attention in deep generative modeling for its mathematical flexibility compared to the Scored-based Generative Model (SGM). However, it remains unclear whether the optimization principle of SB relates to the modern training of deep generative models, which often rely on constructing parameterized loglikelihood objectives.This raises questions on the suitability of SB models as a principled alternative for generative applications. In this work, we present a novel computational framework for likelihood training of SB models grounded on Forward-Backward Stochastic Differential Equations Theory -a mathematical methodology appeared in stochastic optimal control that transforms the optimality condition of SB into a set of SDEs. Crucially, these SDEs can be used to construct the likelihood objectives for SB that, surprisingly, generalizes the ones for SGM as special cases. This leads to a new optimization principle that inherits the same SB optimality yet without losing applications of modern generative training techniques, and we show that the resulting training algorithm achieves comparable results on generating realistic images on MNIST, CelebA, and CIFAR10. * Equal contribution. Order determined by coin flip. See Author Contributions section.

show abstract

Section: Resultsmentioning

confidence: 99%

Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory

Chen¹,

Liu²,

Theodorou³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Note that we did not focus on training our models towards high likelihood. Furthermore, and Huang et al (2020) recently trained augmented Normalizing Flows, which have conceptual similarities with our velocity augmentation. Methods leveraging auxiliary variables similar to our velocities are also used in statistics-such as Hamiltonian Monte Carlo (Neal, 2011)-and have found applications, for instance, in Bayesian machine learning Ding et al, 2014;Shang et al, 2015).…”

Section: Related Workmentioning

confidence: 99%

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Dockhorn¹,

Vahdat²,

Kreis³

2021

Preprint

View full text Add to dashboard Cite

Score-based generative models (SGMs) have demonstrated remarkable synthesis quality. SGMs rely on a diffusion process that gradually perturbs the data towards a tractable distribution, while the generative model learns to denoise. The complexity of this denoising task is, apart from the data distribution itself, uniquely determined by the diffusion process. We argue that current SGMs employ overly simplistic diffusions, leading to unnecessarily complex denoising processes, which limit generative modeling performance. Based on connections to statistical mechanics, we propose a novel critically-damped Langevin diffusion (CLD) and show that CLD-based SGMs achieve superior performance. CLD can be interpreted as running a joint diffusion in an extended space, where the auxiliary variables can be considered "velocities" that are coupled to the data variables as in Hamiltonian dynamics. We derive a novel score matching objective for CLD and show that the model only needs to learn the score function of the conditional distribution of the velocity given data, an easier task than learning scores of the data directly. We also derive a new sampling scheme for efficient synthesis from CLD-based diffusion models. We find that CLD outperforms previous SGMs in synthesis quality for similar network architectures and sampling compute budgets. We show that our novel sampler for CLD significantly outperforms solvers such as Euler-Maruyama. Our framework provides new insights into score-based denoising diffusion models and can be readily used for high-resolution image synthesis. Project page and code: https://nv-tlabs.github.io/CLD-SGM.

show abstract

“…The most basic questions revolve around the representational power of such models. Even the question of universal approximation was only recently answered by three concurrent papers (Huang et al, 2020;Zhang et al, 2020;Koehler et al, 2020)-though in a less-than-satisfactory manner, in light of how normalizing flows are trained. Namely, Huang et al (2020); Zhang et al (2020) show that any (reasonably well-behaved) distribution p, once padded with zeros and treated as a distribution in R d+d ′ , can be arbitrarily closely approximated by an affine coupling flow.…”

Section: Introductionmentioning

confidence: 99%

“…Even the question of universal approximation was only recently answered by three concurrent papers (Huang et al, 2020;Zhang et al, 2020;Koehler et al, 2020)-though in a less-than-satisfactory manner, in light of how normalizing flows are trained. Namely, Huang et al (2020); Zhang et al (2020) show that any (reasonably well-behaved) distribution p, once padded with zeros and treated as a distribution in R d+d ′ , can be arbitrarily closely approximated by an affine coupling flow. While such padding can be operationalized as an algorithm by padding the training image with zeros, it is never done in practice, as it results in an ill-conditioned Jacobian.…”

Section: Introductionmentioning

confidence: 99%

“…Despite the widespread usage of affine couplings, the special structure of the architecture makes understanding their representational power challenging. The question of universal approximation was only recently resolved by three parallel papers (Huang et al, 2020; Zhang et al, 2020;Koehler et al, 2020) who showed reasonably regular distributions can be approximated arbitrarily well using affine couplings-albeit with networks with a nearly-singular Jacobian. As ill-conditioned Jacobians are an obstacle for likelihood-based training, the fundamental question remains: which distributions can be approximated using well-conditioned affine coupling flows?In this paper, we show that any log-concave distribution can be approximated using wellconditioned affine-coupling flows.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Universal Approximation for Log-concave Distributions using Well-conditioned Normalizing Flows

Lee,

Pabbaraju,

Sevekari

et al. 2021

Preprint

View full text Add to dashboard Cite

Normalizing flows are a widely used class of latent-variable generative models with a tractable likelihood. Affine-coupling models (Dinh et al., 2014(Dinh et al., , 2016 are a particularly common type of normalizing flows, for which the Jacobian of the latent-to-observable-variable transformation is triangular, allowing the likelihood to be computed in linear time. Despite the widespread usage of affine couplings, the special structure of the architecture makes understanding their representational power challenging. The question of universal approximation was only recently resolved by three parallel papers (Huang et al., 2020; Zhang et al., 2020;Koehler et al., 2020) who showed reasonably regular distributions can be approximated arbitrarily well using affine couplings-albeit with networks with a nearly-singular Jacobian. As ill-conditioned Jacobians are an obstacle for likelihood-based training, the fundamental question remains: which distributions can be approximated using well-conditioned affine coupling flows?In this paper, we show that any log-concave distribution can be approximated using wellconditioned affine-coupling flows. In terms of proof techniques, we uncover and leverage deep connections between affine coupling architectures, underdamped Langevin dynamics (a stochastic differential equation often used to sample from Gibbs measures) and Hénon maps (a structured dynamical system that appears in the study of symplectic diffeomorphisms). Our results also inform the practice of training affine couplings: we approximate a padded version of the input distribution with iid Gaussians-a strategy which Koehler et al. ( 2020) empirically observed to result in better-conditioned flows, but had hitherto no theoretical grounding. Our proof can thus be seen as providing theoretical evidence for the benefits of Gaussian padding when training normalizing flows. Definition 3. We say that a symmetric matrix is positive semidefinite (PSD) if all of its eigenvalues are non-negative. For symmetric matrices A, B, we write A B if and only if A − B is PSD.Definition 4. Given two probability measures µ, ν over a metric space (M, d), the Wasserstein-1 distance between them, denoted W 1 (µ, ν), is defined aswhere Γ(µ, ν) is the set of couplings, i.e. measures on M × M with marginals µ, ν respectively. For two probability distributions p, q, we denote by W 1 (p, q) the Wasserstein-1 distance between their associated measures. In this paper, we set M = R d and d(x, y) = x − y 2 . Definition 5. Given a distribution q and a compact set C, we denote by q| C the distribution q truncated to the set C. The truncated measure is defined as q| C (A) = 1 q(C) q(A ∩ C). Main resultOur main result states that we can approximate any log-concave distribution in Wasserstein-1 distance by a well-conditioned affine-coupling flow network. Precisely, we show:

show abstract

Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models

Cited by 23 publications

References 0 publications

Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory

Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Universal Approximation for Log-concave Distributions using Well-conditioned Normalizing Flows

Contact Info

Product

Resources

About