Optimization and uncertainty analysis of ODE models using 2nd order adjoint sensitivity analysis

Stapor, Paul; Froehlich, Fabian; Hasenauer, Jan

doi:10.1101/272005

Cited by 4 publications

(8 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To minimize the loss function L with respect to target data, we use the adjoint state of the IDE in Equation 1, cf. [10,31], or backpropagate directly through the IDE solver. The adjoint function is defined as a(t) := ∂L ∂y and its dynamics is determined by another system of IDEs, since it is obtained by differentiating the loss function evaluated on the output of the IDE solver.…”

Section: Learning Dynamics With Integro-differential Equationsmentioning

confidence: 99%

“…In order to update the parameters of the neural networks of the NIDE, we need to consider the augmented system (cf. [10,31]). The augmented state a aug (t) is obtained by considering the augmented IDE, where y aug = [y(t)|θ] is obtained by concatenating the parameters of F and K to y(t).…”

Section: Appendix a Integro-differential Equationsmentioning

confidence: 99%

“…We now consider the dynamics of Equation 1 and derive the corresponding adjoint state, cf. [10,31]. In particular, we want to show that the adjoint function a(t) := ∂L ∂y , where L is the loss function used for training, when f is trivial, satisfies the IDE…”

Section: Appendix B Adjoint Dynamicsmentioning

confidence: 99%

See 2 more Smart Citations

Neural Integro-Differential Equations

Zappala¹,

Fonseca²,

Moberly³

et al. 2022

Preprint

View full text Add to dashboard Cite

Modeling continuous dynamical systems from discretely sampled observations is a fundamental problem in data science. Often, such dynamics are the result of non-local processes that present an integral over time. As such, these systems are modeled with Integro-Differential Equations (IDEs); generalizations of differential equations that comprise both an integral and a differential component. For example, brain dynamics are not accurately modeled by differential equations since their behavior is non-Markovian, i.e. dynamics are in part dictated by history. Here, we introduce the Neural IDE (NIDE), a framework that models ordinary and integral components of IDEs using neural networks. We test NIDE on several toy and brain activity datasets and demonstrate that NIDE outperforms other models, including Neural ODE. These tasks include time extrapolation as well as predicting dynamics from unseen initial conditions, which we test on whole-cortex activity recordings in freely behaving mice. Further, we show that NIDE can decompose dynamics into their Markovian and non-Markovian constituents, via the learned integral operator, which we test on fMRI brain activity recordings of people on ketamine. Finally, the integrand of the integral operator provides a latent space that gives insight into the underlying dynamics, which we demonstrate on wide-field brain imaging recordings. Altogether, NIDE is a novel approach that enables modeling of complex non-local dynamics with neural networks.

show abstract

Section: Learning Dynamics With Integro-differential Equationsmentioning

confidence: 99%

Section: Appendix a Integro-differential Equationsmentioning

confidence: 99%

See 1 more Smart Citation

Neural Integro-Differential Equations

Zappala¹,

Fonseca²,

Moberly³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In order to ensure high calculation accuracy, the step size of ODE Solver is actually set as a small value, which indicates that directly using the gradient back propagation algorithm to calculate the loss function will introduce a large calculation cost. To solve this problem, the adjoint method is proposed in [9] to transform the gradient calculation into an ODE problem, which can be solved by the ODE Solver with low computational cost. To handle the calculation error introduced in the adjoint method in [9], the adaptive checkpoint adjoint method is further proposed in [10] with adding a small amount of storage cost.…”

Section: A Overview Of Neural Ode and Ode-rnnmentioning

confidence: 99%

Mobile MIMO Channel Prediction with ODE-RNN: a Physics-Inspired Adaptive Approach

Xiao¹,

Zhang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Obtaining accurate channel state information (CSI) is crucial and challenging for multiple-input multiple-output (MIMO) wireless communication systems. Conventional channel estimation method cannot guarantee the accuracy of mobile CSI while requires high signaling overhead. Through exploring the intrinsic correlation among a set of historical CSI instances randomly obtained in a certain communication environment, channel prediction can significantly increase CSI accuracy and save signaling overhead. In this paper, we propose a novel channel prediction method based on ordinary differential equation (ODE)-recurrent neural network (RNN) for accurate and flexible mobile MIMO channel prediction. Differing from existing works using sequential network structures for exploring the numerical correlation between observed data, our proposed method tries to represent the implicit physics process of path responses changing by specially designed continuous learning network with ODE structure. Due to the targeted design of learning network, our proposed method fits the mathematics feature of CSI data better and enjoy higher network interpretability. Experimental results show that the proposed learning approach outperforms existing methods, especially for long time interval of the CSI sequence and large channel measurement error.

show abstract

“…For ODEs, we can explicitly write a solver to simulate the dynamical system. Following (Chen et al, 2018), if the ODE has too many parameters or the discretisation appears to affect the quality of the solution, we can learn gradients in a memory-efficient way through the adjoint method (Stapor et al, 2018). The same simulation applies to white-box and black-box methods, and the difference is contained in the function f θ .…”

Section: Simulation: Modified Eulermentioning

confidence: 99%

Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems

Roeder¹,

Grant²,

Phillips³

et al. 2019

Preprint

View full text Add to dashboard Cite

We introduce a flexible, scalable Bayesian inference framework for nonlinear dynamical systems characterised by distinct and hierarchical variability at the individual, group, and population levels. Our model class is a generalisation of nonlinear mixed-effects (NLME) dynamical systems, the statistical workhorse for many experimental sciences. We cast parameter inference as stochastic optimisation of an end-to-end differentiable, block-conditional variational autoencoder. We specify the dynamics of the data-generating process as an ordinary differential equation (ODE) such that both the ODE and its solver are fully differentiable. This model class is highly flexible: the ODE right-hand sides can be a mixture of userprescribed or "white-box" sub-components and neural network or "black-box" sub-components. Using stochastic optimisation, our amortised inference algorithm could seamlessly scale up to massive data collection pipelines (common in labs with robotic automation). Finally, our framework supports interpretability with respect to the underlying dynamics, as well as predictive generalization to unseen combinations of group components (also called "zero-shot" learning). We empirically validate our method by predicting the dynamic behaviour of bacteria that were genetically engineered to function as biosensors.

show abstract

Optimization and uncertainty analysis of ODE models using 2nd order adjoint sensitivity analysis

Cited by 4 publications

References 40 publications

Neural Integro-Differential Equations

Neural Integro-Differential Equations

Mobile MIMO Channel Prediction with ODE-RNN: a Physics-Inspired Adaptive Approach

Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems

Contact Info

Product

Resources

About