The Symplectic Adjoint Method: Memory-Efficient Backpropagation of Neural-Network-Based Differential Equations

Abstract

The combination of neural networks and numerical integration can provide highly accurate models of continuoustime dynamical systems and probabilistic distributions. However, if a neural network is used n times during numerical integration, the whole computation graph can be considered as a network n times deeper than the original. The backpropagation algorithm consumes memory in proportion to the number of uses times of the network size, causing practical difficulties. This is true even if a checkpointing scheme divides the computation graph into subgraphs. Alternatively, the adjoint method obtains a gradient by a numerical integration backward in time; although this method consumes memory only for single-network use, the computational cost of suppressing numerical errors is high. The symplectic adjoint method proposed in this study, an adjoint method solved by a symplectic integrator, obtains the exact gradient (up to rounding error) with memory proportional to the number of uses plus the network size. The theoretical analysis shows that it consumes much less memory than the naive backpropagation algorithm and checkpointing schemes. The experiments verify the theory, and they also demonstrate that the symplectic adjoint method is faster than the adjoint method and is more robust to rounding errors.

Improving neural ordinary differential equations via knowledge distillation

“…Augmented Neural ODEs [14] extended Neural ODEs by using the additional dimensions to learn more complex functions. Matsubara et al [21] improved Neural ODEs via the symplectic adjoint method and demonstrated that the symplectic adjoint method consumes much less memory than the naive backpropagation algorithm and checkpointing schemes, performing faster than the adjoint method.…”

Deep learning‐based time delay estimation for motion compensation in synthetic aperture sonars

“…Neural networks are a type of typical supervised learning methods, which can effectively overcome the limitations of linear models by learning the non-linear mapping between inputs and outputs [14][15][16][17]. Neural networks are composed of some functions associated with a directed graph.…”

“…Backpropagation [15,16] is a widely used training strategy. It utilises the derivative chain rule in order to obtain the weights of each layer.…”

“…Machine learning techniques have been widely used in pattern recognition and automated data processing, including computer vision, speech processing, and some physical science fields [14][15][16][17]. Machine learning methods include three major categories: supervised, unsupervised, and reinforcement learning [17].…”

“…In fact, the time delays in SAS can be understood as a mapping relationship between the echoes and the platform motion, which suggests that supervised learning techniques may be effectively utilised in the time delay estimation of SAS. There are many different forms of mapping in supervised learning, including decision trees, logistic regression, kernel machines, support vector machines (SVMs), Bayesian classifiers, and neural networks [14][15][16][17][18][19][20][21][22][23][24][25][26]. In recent years, deep neural networks have made remarkable progress in the field of supervised learning [14].…”