Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems

Lee, Kookjin; Parish, Eric J.

doi:10.48550/arxiv.2010.14685

Cited by 2 publications

(3 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been many follow-up studies to enhance neural ODEs in different aspects, e.g., enhancing the expressivity of neural ODEs by augmenting extra dimensions in state variables [25], checkpoint methods to mitigate numerical instability and to enhance memory efficiency [26,27,28], allowing network parameters to evolve over time together with hidden states [29,30], and spectrally approximating dynamics by using a set of orthogonal polynomials [31]. Applications of NODEs for learning complex physical processes (e.g., turbulent flow) can be found in [32,33,34].…”

Section: Related Workmentioning

confidence: 99%

Machine learning structure preserving brackets for forecasting irreversible processes

Lee¹,

Trask²,

Stinis³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Forecasting of time-series data requires imposition of inductive biases to obtain predictive extrapolation, and recent works have imposed Hamiltonian/Lagrangian form to preserve structure for systems with reversible dynamics. In this work we present a novel parameterization of dissipative brackets from metriplectic dynamical systems appropriate for learning irreversible dynamics with unknown a priori model form. The process learns generalized Casimirs for energy and entropy guaranteed to be conserved and nondecreasing, respectively. Furthermore, for the case of added thermal noise, we guarantee exact preservation of a fluctuation-dissipation theorem, ensuring thermodynamic consistency. We provide benchmarks for dissipative systems demonstrating learned dynamics are more robust and generalize better than either "black-box" or penalty-based approaches. Background and previous workModeling time-series data as a solution to a dynamical system with learnable dynamics has been shown to be effective in both data-driven modeling for physical systems and traditional machine learning (ML) tasks. Broadly, it has been observed that imposition of physics-based structure leads to more robust architectures which generalize well [1]. On one end of the spectrum of inductive biases, universal differential equations (UDE) [2] assume an a priori known model form, thus imposing the strongest bias. On the other, neural ordinary differential equations (NODEs) [3] assume a completely black-box model form with minimal bias.Many recent approaches have turned to structure preserving models of reversible dynamics to obtain an inductive bias that lies in between [4,5,6,7,8]. One may use black-box deep neural networks (DNNs) to learn an energy of a system with unknown model form, while the algebraic structure of Hamiltonian/Lagrangian dynamics provides a flow map which conserves energy. Typically, the learned flow map has symplectic structure so that phase space trajectories are conserved. In classification problems, this mitigates the vanishing/exploding gradient problem and improves accuracy [9]; in physics, this guarantees that extrapolated states are physically realizable [10].Such approaches are only appropriate for reversible systems lacking friction or dissipation. In the physics literature, the theory of metriplectic dynamical systems provides a generalization of the Poisson brackets of Hamiltonian/Lagrangian mechanics which model not just a conserved energy, but generalized Casimirs such as entropy [11,12]. Physical systems which can be cast in this framework Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Machine learning structure preserving brackets for forecasting irreversible processes

Lee¹,

Trask²,

Stinis³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Guen et al [11] combine neural ODEs with approximate physical models to forecast the evolution of dynamical systems. Inspired by computational science and engineering, Lee and Parish [17] propose the parameterized neural ODEs that extend NODEs to have a set of input parameters, for learning multiple complex dynamics altogether.…”

Section: Related Workmentioning

confidence: 99%

“…Learning dynamics governed by differential equations is crucial for predicting and controlling the systems in science and engineering such as predicting future movements of planets in physics, protein structure prediction [30], evolution of fluid flow [27] and many other applications [23]. The recently proposed neural ordinary differential equations (Neural ODEs) [1], a deep learning model integrated with differential equations shows great promise in scientific field [11,20,31,17,2]. The continuous nature of NODEs and their differential equation structure of the hypothesis have made them particularly suitable for learning the dynamics of complex physical systems.…”

Section: Introductionmentioning

confidence: 99%

Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics

Gong¹,

Meng²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning dynamics governed by differential equations is crucial for predicting and controlling the systems in science and engineering. Neural Ordinary Differential Equation (NODE), a deep learning model integrated with differential equations, learns the dynamics directly from the samples on the trajectory and shows great promise in the scientific field. However, the training of NODE highly depends on the numerical solver, which can amplify numerical noise and be unstable, especially for ill-conditioned dynamical systems. In this paper, to reduce the reliance on the numerical solver, we propose to enhance the supervised signal in learning dynamics. Specifically, beyond learning directly from the trajectory samples, we pre-train a neural differential operator (NDO) to output an estimation of the derivatives to serve as an additional supervised signal. The NDO is pre-trained on a class of symbolic functions, and it learns the mapping between the trajectory samples of these functions to their derivatives. We provide theoretical guarantee on that the output of NDO can well approximate the ground truth derivatives by proper tuning the complexity of the library. To leverage both the trajectory signal and the estimated derivatives from NDO, we propose an algorithm called NDO-NODE, in which the loss function contains two terms: the fitness on the true trajectory samples and the fitness on the estimated derivatives that are output by the pre-trained NDO. Experiments on various of dynamics show that our proposed NDO-NODE can consistently improve the forecasting accuracy.

show abstract

Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems

Cited by 2 publications

References 57 publications

Machine learning structure preserving brackets for forecasting irreversible processes

Machine learning structure preserving brackets for forecasting irreversible processes

Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics

Contact Info

Product

Resources

About