Variational Inference with Normalizing Flows

Rezende, Danilo Jimenez; Mohamed, Shakir

doi:10.48550/arxiv.1505.05770

Cited by 221 publications

(309 citation statements)

References 0 publications

Supporting

Mentioning

307

Contrasting

Unclassified

Order By: Relevance

“…However, most methods do not model data density on the manifold and are thus not applicable for the same purposes as the models introduced in our work. Popular density-based probabilistic models such as Variational Auto-Encoder [21] or Flow variants [20,35,11,26,38] assume the data distribution is a.c. and as such cannot model distributions that lie on a low-dimensional manifold. A number of recent works do however introduce normalizing flows on manifolds such as Riemannian Flows [14,36] and M-Flow [5].…”

Section: Comparions With Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Spread Flows for Manifold Modelling

Zhang¹,

Sun²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Flow-based generative models typically define a latent space with dimensionality identical to the observational space. In many problems, however, the data does not populate the full ambient data-space that they natively reside in, rather inhabiting a lower-dimensional manifold. In such scenarios, flow-based models are unable to represent data structures exactly as their density will always have support off the data manifold, potentially resulting in degradation of model performance. In addition, the requirement for equal latent and data space dimensionality can unnecessarily increase complexity for contemporary flow models. Towards addressing these problems, we propose to learn a manifold prior that affords benefits to both sample generation and representation quality. An auxiliary benefit of our approach is the ability to identify the intrinsic dimension of the data distribution. * The work was done during an internship in Huawei Noah's Ark Lab. 2 The distribution is absolutely continuous with respect to the Lebesgue measure. In this case, it has a density function, see [12].Preprint. Under review.

show abstract

Section: Comparions With Related Workmentioning

confidence: 99%

“…Normalizing flows [35,22] have shown considerable potential for the task of modelling and inferring expressive distributions through the learning of well-specified probabilistic models. Specifically, assume an absolutely continuous 2 (a.c.) random variable (r.v.)…”

Section: Introductionmentioning

confidence: 99%

Spread Flows for Manifold Modelling

Zhang¹,

Sun²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Normalizing Flows are a class of generative models where an 'simple' paramaterizable base distribution is transformed into a more complex approximation for the posterior distribution (Rezende and Mohamed 2015). This transformation is achieved by passing the base distribution through a series of invertible and bijective mappings.…”

Section: Flow-based Generative Modelsmentioning

confidence: 99%

“…Other popular approaches in this domain include fully visible belief networks like NADE (Uria et al 2016), MADE (Kumar et al 2016), PixelCNN, PixelRNN Van den Oord et al 2016) and WaveNet . Normalizing Flows (Rezende and Mohamed 2015) fall under the category broadly known as change of variable models, which employ the theorem to transform a simple parameterizable base distribution into some complex approximation of the posterior distribution. In contrast to GANs and VAEs, normalizing flows are an attractive class due to their ability to obtain tractable density estimates.…”

Section: Normalizing Flowsmentioning

confidence: 99%

See 1 more Smart Citation

Attentive Contractive Flow: Improved Contractive Flows with Lipschitz-constrained Self-Attention

Mukherjee¹,

Patro²,

Sidheekh³

et al. 2021

Preprint

View full text Add to dashboard Cite

Normalizing flows provide an elegant method for obtaining tractable density estimates from distributions by using invertible transformations. The main challenge is to improve the expressivity of the models while keeping the invertibility constraints intact. We propose to do so via the incorporation of localized self-attention. However, conventional self-attention mechanisms don't satisfy the requirements to obtain invertible flows and can't be naively incorporated into normalizing flows. To address this, we introduce a novel approach called Attentive Contractive Flow (ACF) which utilizes a special category of flow-based generative models -contractive flows. We demonstrate that ACF can be introduced into a variety of state of the art flow models in a plug-and-play manner. This is demonstrated to not only improve the representation power of these models (improving on the bits per dim metric), but also to results in significantly faster convergence in training them. Qualitative results, including interpolations between test images, demonstrate that samples are more realistic and capture local correlations in the data well. We evaluate the results further by performing perturbation analysis using AWGN demonstrating that ACF models (especially the dotproduct variant) show better and more consistent resilience to additive noise.

show abstract

Deep generative models for vehicle speed trajectories

Behnia,

Karbowski,

Sokolov

2023

Appl Stoch Models Bus & Ind

View full text Add to dashboard Cite

Generating realistic vehicle speed trajectories is a crucial component in evaluating vehicle fuel economy and in predictive control of self‐driving cars. Traditional generative models rely on Markov chain methods and can produce accurate synthetic trajectories but are subject to the curse of dimensionality. They do not allow to include conditional input variables into the generation process. In this paper, we show how extensions to deep generative models allow accurate and scalable generation. Proposed architectures involve recurrent and feed‐forward layers and are trained using adversarial techniques. Our models are shown to perform well on generating vehicle trajectories using a model trained on GPS data from Chicago metropolitan area.

show abstract

Variational Inference with Normalizing Flows

Cited by 221 publications

References 0 publications

Spread Flows for Manifold Modelling

Spread Flows for Manifold Modelling

Attentive Contractive Flow: Improved Contractive Flows with Lipschitz-constrained Self-Attention

Deep generative models for vehicle speed trajectories

Contact Info

Product

Resources

About