Deep learning algorithm for data-driven simulation of noisy dynamical system

Yeo, Kyongmin; Melnyk, Igor

doi:10.1016/j.jcp.2018.10.024

Cited by 99 publications

(66 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep neural networks are capable of approximating nonlinear dynamical systems as shown in many studies 102,103,106,123 . The general nonlinear dynamical system can be presented by an equation of the form…”

Section: Learning Frameworkmentioning

confidence: 99%

A deep learning enabler for nonintrusive reduced order modeling of fluid flows

et al. 2019

View full text Add to dashboard Cite

In this paper, we introduce a modular deep neural network (DNN) framework for data-driven reduced order modeling of dynamical systems relevant to fluid flows. We propose various deep neural network architectures which numerically predict evolution of dynamical systems by learning from either using discrete state or slope information of the system. Our approach has been demonstrated using both residual formula and backward difference scheme formulas. However, it can be easily generalized into many different numerical schemes as well. We give a demonstration of our framework for three examples: (i) Kraichnan-Orszag system, an illustrative coupled nonlinear ordinary differential equations, (ii) Lorenz system exhibiting chaotic behavior, and (iii) a non-intrusive model order reduction framework for the two-dimensional Boussinesq equations with a differentially heated cavity flow setup at various Rayleigh numbers. Using only snapshots of state variables at discrete time instances, our data-driven approach can be considered truly non-intrusive, since any prior information about the underlying governing equations is not required for generating the reduced order model. Our a posteriori analysis shows that the proposed data-driven approach is remarkably accurate, and can be used as a robust predictive tool for non-intrusive model order reduction of complex fluid flows.

show abstract

Section: Learning Frameworkmentioning

confidence: 99%

A deep learning enabler for nonintrusive reduced order modeling of fluid flows

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Here, θ j is the j-th column of θ. Note that after a proper rescaling, the summations in the original ESN formulation (7) are time averaging, while the summations in (19) correspond to an ensemble averaging, or a Monte Carlo sampling. Hence, when the dynamical system is ergodic, the time series data is missing in random, and the number of observation, ωT , is large enough, the linear map of the fixed-point method, θ k , approaches to θ * as the fixed-point iteration converges.…”

Section: Fixed-point Esn For Sparse Observationmentioning

confidence: 99%

“…Over the last decade, artificial neural network (ANN) has been extensively studied due to its strength in identifying complex nonlinear structure in data [1,2,3]. Recent studies on the application of ANN on the data-driven modeling of physical systems have shown promising results [4,5,6,7]. In driven reconstruction of the missing information by using recurrent neural networks (RNN).…”

Section: Introductionmentioning

confidence: 99%

Data-driven reconstruction of nonlinear dynamics from sparse observation

Yeo

2019

Journal of Computational Physics

Self Cite

View full text Add to dashboard Cite

We present a data-driven model to reconstruct nonlinear dynamics from a very sparse times series data, which relies on the strength of the echo state network (ESN) in learning nonlinear representation of data. With an assumption of the universal function approximation capability of ESN, it is shown that the reconstruction problem can be formulated as a fixed-point problem, in which the trajectory of the dynamical system is a fixed point of the ESN. An under-relaxed fixed-point iteration is proposed to reconstruct the nonlinear dynamics from a sparse observation. The proposed fixed-point ESN is tested against both univariate and multivariate chaotic dynamical systems by randomly removing up to 95% of the data. It is shown that the fixed-point ESN is able to reconstruct the complex dynamics from only 5 ∼ 10% of the data. For a relatively simple non-chaotic dynamical system, the numerical experiments on a forced van der Pol oscillator show that it is possible to reconstruct the nonlinear dynamics from only 1∼2% of the data. particular, in [8,9], it is shown that ANN is capable of predicting the dynamics of a chaotic attractor. When training an ANN for the modeling of a dynamical system, it is typical to use a sufficiently long time series data, so that the ANN fully explores the phase space. Given the huge amount of training data, it is unclear whether the long-term forecast capability of ANN is due to a high dimensional interpolation in a phase space, similar to the delay-coordinate embedding [10,11], or because ANN really learns the nonlinear dynamics. As a first step to answer this question, we first consider the problem, where only a partial observation of a time series data is available. We investigate if ANN can reconstruct the nonlinear dynamics from the incomplete information.Reconstructing dynamics from an incomplete observation has practical applications in many physical, biological, and engineering problems, where the state of the system is accessible only through a sensor network. For example, in geophysics, it is not uncommon to find time series data, which contains a large amount of missing information or is irregularly sampled, due to sensor malfunction, atmospheric conditions, or physical limitations of the sensors [12,13,14]. When a priori knowledge on the physical system is available, one of the standard approaches to reconstruct the nonlinear dynamics from an incomplete data set is to design a statistical model that incorporates the physical knowledge [15,16,17]. In [18], it is shown that a high-resolution temporal dynamics can be reconstructed from a low-resolution time series data, which has a 10 times lower time resolution, by using only a subset of the governing equations. Incorporating the physics knowledge on the training of ANN, [6] proposed "physics informed neural network", which can recover the complex nonlinear dynamics from only a small fraction of the data.It is challenging to reconstruct nonlinear dynamics without having any prior knowledge on the underlying process. There are three ...

show abstract

“…Since a number of authors have begun to consider the use of machine/deep learning for problems in traditional computational physics, see e.g. [1,2,3,4,5,6,7,8,9,10,11,12], we are motivated to consider methodologies that constrain the interpolatory results of a network to be contained within a physically admissible region. Quite recently, [13] proposed adding physical constraints to generative adversarial networks (GANs) also considering projection as we do, while stressing the interplay between scientific computing and machine learning; we refer the interested reader to their work for even more motivation for such approaches.…”

Section: Introductionmentioning

confidence: 99%

Coercing machine learning to output physically accurate results

Geng

Johnson

Fedkiw

2020

Journal of Computational Physics

View full text Add to dashboard Cite

Many machine/deep learning artificial neural networks are trained to simply be interpolation functions that map input variables to output values interpolated from the training data in a linear/nonlinear fashion. Even when the input/output pairs of the training data are physically accurate (e.g. the results of an experiment or numerical simulation), interpolated quantities can deviate quite far from being physically accurate. Although one could project the output of a network into a physically feasible region, such a postprocess is not captured by the energy function minimized when training the network; thus, the final projected result could incorrectly deviate quite far from the training data. We propose folding any such projection or postprocess directly into the network so that the final result is correctly compared to the training data by the energy function. Although we propose a general approach, we illustrate its efficacy on a specific convolutional neural network that takes in human pose parameters (joint rotations) and outputs a prediction of vertex positions representing a triangulated cloth mesh. While the original network outputs vertex positions with erroneously high stretching and compression energies, the new network trained with our physics prior remedies these issues producing highly improved results. there will be large errors inf (w, x T ) when compared to y T . On the other hand, although one could create a network with many degrees of freedom in order to capture y T =f (w, x T ) as accurately as desired, even exactly,f (w, x) could oscillate wildly and inaccurately when x is not equal to x T , i.e. overfitting. See, e.g. [14,15,16,17]. One needs to take great care when designing the network architecture in order to avoid underfitting while still allowing for enough regularization to also avoid overfitting. Likewise, the form of the energy function and nature of the numerical optimization techniques also need careful consideration. Some of the most popular methods include variants of BFGS [18,19,20] and a number of methods based on gradient descent [21,22] (see also [23] and the references therein) or interpreting gradient descent as a numerical approximation to an ordinary differential equation to be solved via various approaches motivated by order of accuracy [24,25] and adaptive time-stepping [26,27,28,29,30].Devising a network architecture with enough representative capability to alleviate underfitting while still being amenable to the regularization required to avoid overfitting, and subsequently applying numerical optimization techniques to an adequately designed energy in order to find reasonable parameters w is a quite difficult and mostly experimental endeavour. Thus, much of the progress made by the community emanates from the laborious creation of data sets that many researchers can consider in order to design network architectures and find suitable parameters w, see e.g. [31]. This is typically driven by a community (rather than an individual or group) effort, and state-of-the-art r...

show abstract

Deep learning algorithm for data-driven simulation of noisy dynamical system

Cited by 99 publications

References 37 publications

A deep learning enabler for nonintrusive reduced order modeling of fluid flows

A deep learning enabler for nonintrusive reduced order modeling of fluid flows

Data-driven reconstruction of nonlinear dynamics from sparse observation

Coercing machine learning to output physically accurate results

Contact Info

Product

Resources

About