Neural Integro-Differential Equations

Abstract

Modeling continuous dynamical systems from discretely sampled observations is a fundamental problem in data science. Often, such dynamics are the result of non-local processes that present an integral over time. As such, these systems are modeled with Integro-Differential Equations (IDEs); generalizations of differential equations that comprise both an integral and a differential component. For example, brain dynamics are not accurately modeled by differential equations since their behavior is non-Markovian, i.e. dynamics are in part dictated by history. Here, we introduce the Neural IDE (NIDE), a framework that models ordinary and integral components of IDEs using neural networks. We test NIDE on several toy and brain activity datasets and demonstrate that NIDE outperforms other models, including Neural ODE. These tasks include time extrapolation as well as predicting dynamics from unseen initial conditions, which we test on whole-cortex activity recordings in freely behaving mice. Further, we show that NIDE can decompose dynamics into their Markovian and non-Markovian constituents, via the learned integral operator, which we test on fMRI brain activity recordings of people on ketamine. Finally, the integrand of the integral operator provides a latent space that gives insight into the underlying dynamics, which we demonstrate on wide-field brain imaging recordings. Altogether, NIDE is a novel approach that enables modeling of complex non-local dynamics with neural networks.

Neural Integro-Differential Equations

“…Next, to inspect the extent to which NIDE can generalize to new initial conditions, we consider the model trained on the 4D curves dataset and evaluate it on curves from new initial conditions that have not been seen during training. We find that NIDE yields lower MSE for predicted dynamics from initial conditions than NODE, as shown in Table 5 in Zappala et al (2022) per extrapolated time point.…”

“…We show that when the dynamics are generated by an IDE with sufficiently complex non-local properties, NODE is not capable of properly fitting the data, but NIDE is. Details about the architecture of the model used in each task are provided in Table 2 Zappala et al (2022).…”

“…Notice that the functions α(t) and β(t) are arbitrary and, therefore, for each t ∈ [t 0 , t 1 ], the derivative of the function y(t), dy dt , depends on values of y(t) both in the past, the present, and the future of the dynamics. Common choices of α and β are α(t) = 0 and β(t) = t (Volterra IDEs) or α(t) = a and β(t) = b (Fredholm IDEs), see Zappala et al (2022) Appendix A. Effectively, this means that the RHS of Equation 1 is a functional of y rather than a function.…”

“…We find R 2 = 0.991 ± 0.01 (mean±std) for the whole fitting, R 2 = 0.957 ± 0.03 for the instantaneous part, and R 2 = 0.943 ± 0.05 for the integral part. An example of a reconstructed decomposition is shown in Figure 4 of Zappala et al (2022). Table 1: Average MSE for NIDE (ours) and NODE on fitting data sampled from an IDE generated self-intersecting spiral (lower is better).…”

Numerical analysis of finite difference schemes arising from time-memory partial integro-differential equations

“…There are enormous physical, economic, and biological phenomena modeled based on these equations. For instance, neural IDEs are used to model brain dynamics, and it has been investigated based on the deep learning framework, see Zappala et al [6]. The population dynamics are modeled based on nonlinear PIDEs and solved using the pseudo-spectral technique [2].…”