Neural Controlled Differential Equations for Irregular Time Series

Abstract

Neural ordinary differential equations are an attractive option for modelling temporal dynamics. However, a fundamental issue is that the solution to an ordinary differential equation is determined by its initial condition, and there is no mechanism for adjusting the trajectory based on subsequent observations. Here, we demonstrate how this may be resolved through the well-understood mathematics of controlled differential equations. The resulting neural controlled differential equation model is directly applicable to the general setting of partiallyobserved irregularly-sampled multivariate time series, and (unlike previous work on this problem) it may utilise memory-efficient adjoint-based backpropagation even across observations. We demonstrate that our model achieves state-of-the-art performance against similar (ODE or RNN based) models in empirical studies on a range of datasets. Finally we provide theoretical results demonstrating universal approximation, and that our model subsumes alternative ODE models.

Stability of Deep Neural Networks via discrete rough paths

“…( 20) and ( 21) respectively, with y 0 " y, x 0 " x. The following result draws on ideas by Kidger, Morrill, Foster and Lyons [15].…”

Neural Piecewise-Constant Delay Differential Equations

“…Specifically, when we consider the continuous-depth neural network dx(t) dt = f (x(t), u(t), t), u(t) is regarded as a controller. For example, u(t) treated as constant parameters yields the network frameworks proposed in (Chen et al 2018), u(t) as a data-driven controller yields a framework in (Massaroli et al 2020), and u(t) as other forms of controllers brings more fruitful network structures (Chalvidal et al 2020;Li et al 2020;Kidger et al 2020;Zhu, Guo, and Lin 2021). Here, the mission of this work is to design a delayed feedback controller for rendering a continuous-depth neural network more effectively in coping with synthetic or/and real-world datasets.…”

Characteristic Neural Ordinary Differential Equations

“…But the authors did not consider using adjoint-based training. Kidger et al [2020] expanded to the controlled differential equations motivated as a continuous recurrent neural network framework.…”

“…However, since NODEs can only represent solutions to ODEs, the class of functions is somewhat limited and may not apply to more general problems that do not have smooth and one-to-one mappings. To address this limitation, a series of analyses based on methods from numerical analysis and dynamical systems have been employed to enhance the representation capabilities of NODEs, such as the technique of controlled differential equations [Kidger et al, 2020], learning higher-order ODEs [Massaroli et al, 2021], augmenting dynamics [Dupont et al, 2019], and considering dynamics with delay terms [Zhu et al, 2021]. Moreover, certain works consider generalizing the ODE case to partial differential equations (PDEs), such as in Sun et al [2019].…”