Multiple Shooting for Training Neural Differential Equations on Time Series

Turan, Evren M.; Jäschke, Johannes

doi:10.1109/lcsys.2021.3135835

Cited by 18 publications

(6 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is worthwhile to note that increasing the number of layers and nodes (the layer size was increased from 5 to 9 and the node size was increased from [5,30] to [10,70]) in the black box model does not significantly contribute to the fitting result. This problem is addressed in the previous literature (Turan et al 2021). Multiple shooting is a potential solution to improve performance.…”

Section: Resultsmentioning

confidence: 99%

Physics‐Informed Gas Lifting Oil Well Modelling using Neural Ordinary Differential Equations

Ban,

Pfeiffer

2023

INCOSE International Symp

View full text Add to dashboard Cite

Modelling of oil well systems is important for a wide range of petroleum scientific and oil industrial processes. Considering the uncertainty of the measurements and the demand for empirical knowledge, a purely first‐principle model and a black‐box model based on data are not sufficient for accurately describing an oil well system. Thus, there is a growing body of literature that recognizes the importance of data‐driven methods combined with physical knowledge. However, the application of combination methods for dynamic nonlinear systems is still challenging. In this work, we demonstrate the application of a physics‐informed neural network to a gas lifting oil well system. The neural ordinary differential equation is the main tool for the modeling and the simulation is examined in Julia programming language. The advantage and drawbacks of the physics‐informed data‐driven method are analyzed.

show abstract

Section: Resultsmentioning

confidence: 99%

Physics‐Informed Gas Lifting Oil Well Modelling using Neural Ordinary Differential Equations

Ban,

Pfeiffer

2023

INCOSE International Symp

View full text Add to dashboard Cite

show abstract

“…This could also explain the reduced forecasting accuracy of UDEs compared to other models, as well as why incorporating time as an input improved the accuracy of UDEs. This could be addressed in a future work using alternative training routines such as the multiple shooting method which trains the NODE model starting from different initial conditions found in the data [55]. Other studies have suggested that regularization of the neural network is not necessary to prevent overfitting, as the known dynamics of the UDE implicitly regularize the neural network [56].…”

Section: Discussionmentioning

confidence: 99%

Using neural ordinary differential equations to predict complex ecological dynamics from population density data

Arroyo-Esquivel,

Klausmeier,

Litchman

2024

J. R. Soc. Interface.

View full text Add to dashboard Cite

Simple models have been used to describe ecological processes for over a century. However, the complexity of ecological systems makes simple models subject to modelling bias due to simplifying assumptions or unaccounted factors, limiting their predictive power. Neural ordinary differential equations (NODEs) have surged as a machine-learning algorithm that preserves the dynamic nature of the data (Chen et al. 2018 Adv. Neural Inf. Process. Syst. ). Although preserving the dynamics in the data is an advantage, the question of how NODEs perform as a forecasting tool of ecological communities is unanswered. Here, we explore this question using simulated time series of competing species in a time-varying environment. We find that NODEs provide more precise forecasts than autoregressive integrated moving average (ARIMA) models. We also find that untuned NODEs have a similar forecasting accuracy to untuned long-short term memory neural networks and both are outperformed in accuracy and precision by empirical dynamical modelling . However, we also find NODEs generally outperform all other methods when evaluating with the interval score, which evaluates precision and accuracy in terms of prediction intervals rather than pointwise accuracy. We also discuss ways to improve the forecasting performance of NODEs. The power of a forecasting tool such as NODEs is that it can provide insights into population dynamics and should thus broaden the approaches to studying time series of ecological communities.

show abstract

“…Therefore, we adopt the multiple shooting technique (MS) [38]. In MS, the time interval ( t 0 , t m ) is partitioned into different segments.…”

Section: Methodsmentioning

confidence: 99%

Robust parameter estimation and identifiability analysis with Hybrid Neural Ordinary Differential Equations in Computational Biology

Giampiccolo,

Reali,

Fochesato

et al. 2024

Preprint

View full text Add to dashboard Cite

Parameter estimation is one of the central problems in computational modeling of biological systems. Typically, scientists must fully specify the mathematical structure of the model, often expressed as a system of ordinary differential equations, to estimate the parameters. This process poses significant challenges due to the necessity for a detailed understanding of the underlying biological mechanisms. In this paper, we present an approach for estimating model parameters and assessing their identifiability in situations where only partial knowledge of the system structure is available. The partially known model is extended into a system of Hybrid Neural Ordinary Differential Equations, which captures the unknown portions of the system using neural networks.Integrating neural networks into the model structure introduces two primary challenges for parameter estimation: the need to globally explore the search space while employing gradient-based optimization, and the assessment of parameter identifiability, which may be hindered by the expressive nature of neural networks. To overcome the first issue, we treat biological parameters as hyperparameters in the extended model, exploring the parameter search space during hyperparameter tuning. The second issue is then addressed by ana posteriorianalysis of parameter identifiability, computed by introducing a variant of a well-established approach for mechanistic models. These two components are integrated into an end-to-end pipeline that is thoroughly described in the paper. We assess the effectiveness of the proposed workflow on test cases derived from three different benchmark models. These test cases have been designed to mimic real-world conditions, including the presence of noise in the training data and various levels of data availability for the system variables.Author summaryParameter estimation is a central challenge in modeling biological systems. Typically, scientists calibrate the parameters by aligning model predictions with measured data once the model structure is defined. Our paper introduces a workflow that leverages the integration between mechanistic modeling and machine learning to estimate model parameters when the model structure is not fully known. We focus mainly on analyzing the identifiability of the model parameters, which measures how confident we can be in the parameter estimates given the available experimental data and partial mechanistic understanding of the system. We assessed the effectiveness of our approach in variousin silicoscenarios. Our workflow represents a first step to adapting traditional methods used in fully mechanistic models to the scenario of hybrid modeling.

show abstract

Multiple Shooting for Training Neural Differential Equations on Time Series

Cited by 18 publications

References 22 publications

Physics‐Informed Gas Lifting Oil Well Modelling using Neural Ordinary Differential Equations

Physics‐Informed Gas Lifting Oil Well Modelling using Neural Ordinary Differential Equations

Using neural ordinary differential equations to predict complex ecological dynamics from population density data

Robust parameter estimation and identifiability analysis with Hybrid Neural Ordinary Differential Equations in Computational Biology

Contact Info

Product

Resources

About