Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs

Abstract

Learning neural ODEs often requires solving very stiff ODE systems, primarily using explicit adaptive step size ODE solvers. These solvers are computationally expensive, requiring the use of tiny step sizes for numerical stability and accuracy guarantees. This paper considers learning neural ODEs using implicit ODE solvers of different orders leveraging proximal operators. The proximal implicit solver consists of inner-outer iterations: the inner iterations approximate each implicit update step using a fast optimization algorithm, and the outer iterations solve the ODE system over time. The proximal implicit ODE solver guarantees superiority over explicit solvers in numerical stability and computational efficiency. We validate the advantages of proximal implicit solvers over existing popular neural ODE solvers on various challenging benchmark tasks, including learning continuous-depth graph neural networks and continuous normalizing flows.

Data-driven optimal control with neural network modeling of gradient flows

“…However, to address the numerical errors inherent in this approach, several techniques have been proposed. These include the checkpoint method [20,66], the asynchronous leapfrog method [67], the symplectic adjoint method [46], interpolation techniques [15], and the utilizaiton of proximal implicit solvers [4].…”