Stochastic Normalizing Flows

Hodgkinson, Liam; van der Heide, Chris; Roosta, Fred; Mahoney, Michael W.

doi:10.48550/arxiv.2002.09547

Cited by 7 publications

(11 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These continuous-depth models give rise to vector field representations and a set of functions which were not possible to generate before. These capabilities enabled flexible density estimation (Dupont et al, 2019, Grathwohl et al, 2018, Hodgkinson et al, 2020, Mathieu and Nickel, 2020, Yang et al, 2019, as well as performant modeling of sequential and irregularly-sampled data (Erichson et al, 2021, Gholami et al, 2019, Lechner and Hasani, 2020, Rubanova et al, 2019. In this paper, we showed how to relax the need for an ODE-solver to realize an expressive continuous-time neural network model for challenging time-series problems.…”

Section: Related Workmentioning

confidence: 93%

Closed-form Continuous-time Neural Models

Hasani¹,

Lechner²,

Amini³

et al. 2021

Preprint

View full text Add to dashboard Cite

Continuous-depth neural models, where the derivative of the model's hidden state is defined by a neural network, have enabled strong sequential data processing capabilities. However, these models rely on advanced numerical differential equation (DE) solvers resulting in a significant overhead both in terms of computational cost and model complexity. In this paper, we present a new family of models, termed Closed-form Continuous-depth (CfC) networks, that are simple to describe and at least one order of magnitude faster while exhibiting equally strong modeling abilities compared to their ODE-based counterparts. The models are hereby derived from the analytical closed-form solution of an expressive subset of time-continuous models, thus alleviating the need for complex DE solvers all together. In our experimental evaluations, we demonstrate that CfC networks outperform advanced, recurrent models over a diverse set of time-series prediction tasks, including those with long-term dependencies and irregularly sampled data. We believe our findings open new opportunities to train and deploy rich, continuous neural models in resource-constrained settings, which demand both performance and efficiency.

show abstract

Section: Related Workmentioning

confidence: 93%

Closed-form Continuous-time Neural Models

Hasani¹,

Lechner²,

Amini³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Exploiting the connections between ODE-Nets and the theory of dynamical systems and control is an active area of research [50,53,68], that has also motivated the development of more memory efficient training strategies [13,19,20,54,72] for ODE-Nets. Other research fronts include normalizing flows [21,36,65], and stochastic differential equations [23,32,34,42,45].…”

Section: Related Workmentioning

confidence: 99%

“…We consider a model that has 3 ODE-blocks. The units within the three blocks have an increasing number of channels, [16,32,64].…”

Section: Compressing Deep Ode-nets For Image Classification Tasksmentioning

confidence: 99%

Stateful ODE-Nets using Basis Function Expansions

Queiruga¹,

Erichson²,

Hodgkinson³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The recently-introduced class of ordinary differential equation networks (ODE-Nets) establishes a fruitful connection between deep learning and dynamical systems. In this work, we reconsider formulations of the weights as continuous-depth functions using linear combinations of basis functions. This perspective allows us to compress the weights through a change of basis, without retraining, while maintaining near state-of-the-art performance. In turn, both inference time and the memory footprint are reduced, enabling quick and rigorous adaptation between computational environments. Furthermore, our framework enables meaningful continuous-in-time batch normalization layers using function projections. The performance of basis function compression is demonstrated by applying continuous-depth models to (a) image classification tasks using convolutional units and (b) sentence-tagging tasks using transformer encoder units.

show abstract

“…A related work may be found in [64], where equations for time-varying cumulative distribution functions could directly be learned given access to a fine-scaled simulator. Another relevant work in the literature was presented in [21] where SDEs are utilized to parameterize the transformation of a sample from a latent (generally a time-invariant multivariate isotropic Gaussian) to a time-varying target density. It is important to clarify here that concatenated transformations between supports can also be devised as flow maps governed by ordinary differential equations (ODEs) or SDEs (like in [21]).…”

Section: Introductionmentioning

confidence: 99%

“…Another relevant work in the literature was presented in [21] where SDEs are utilized to parameterize the transformation of a sample from a latent (generally a time-invariant multivariate isotropic Gaussian) to a time-varying target density. It is important to clarify here that concatenated transformations between supports can also be devised as flow maps governed by ordinary differential equations (ODEs) or SDEs (like in [21]). Note that the solutions of these equations are obtained through numerical discretization in fictitious time and must not be confused with the physical time evolution of the target densities themselves; evolution in this fictitious time is then learned and deployed to construct each reference-to-target transformation.…”

Section: Introductionmentioning

confidence: 99%

Learning the temporal evolution of multivariate densities via normalizing flows

Lu,

Maulik,

Gao

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work, we propose a method to learn probability distributions using sample path data from stochastic differential equations. Specifically, we consider temporally evolving probability distributions (e.g., those produced by integrating local or nonlocal Fokker-Planck equations). We analyze this evolution through machine learning assisted construction of a time-dependent mapping that takes a reference distribution (say, a Gaussian) to each and every instance of our evolving distribution. If the reference distribution is the initial condition of a Fokker-Planck equation, what we learn is the time-T map of the corresponding solution. Specifically, the learned map is a normalizing flow that deforms the support of the reference density to the support of each and every density snapshot in time. We demonstrate that this approach can learn solutions to non-local Fokker-Planck equations, such as those arising in systems driven by both Brownian and Lévy noise. We present examples with two-and three-dimensional, uni-and multimodal distributions to validate the method.

show abstract

Stochastic Normalizing Flows

Cited by 7 publications

References 23 publications

Closed-form Continuous-time Neural Models

Closed-form Continuous-time Neural Models

Stateful ODE-Nets using Basis Function Expansions

Learning the temporal evolution of multivariate densities via normalizing flows

Contact Info

Product

Resources

About