KAM Theory Meets Statistical Learning Theory: Hamiltonian Neural Networks with Non-zero Training Loss

Abstract

Many physical phenomena are described by Hamiltonian mechanics using an energy function (Hamiltonian). Recently, the Hamiltonian neural network, which approximates the Hamiltonian by a neural network, and its extensions have attracted much attention. This is a very powerful method, but theoretical studies are limited. In this study, by combining the statistical learning theory and KAM theory, we provide a theoretical analysis of the behavior of Hamiltonian neural networks when the learning error is not completely zero. A Hamiltonian neural network with non-zero errors can be considered as a perturbation from the true dynamics, and the perturbation theory of the Hamilton equation is widely known as KAM theory. To apply KAM theory, we provide a generalization error bound for Hamiltonian neural networks by deriving an estimate of the covering number of the gradient of the multi-layer perceptron, which is the key ingredient of the model. This error bound gives a sup-norm bound on the Hamiltonian that is required in the application of KAM theory.

Unifying physical systems' inductive biases in neural ODE using dynamics constraints

“…[16] further improves the model by introducing a learnable 1 × 1 invertible convolution to replace the fixed permutation of channel dimension used in [15]. In connection with NODE, [17] leverages INNs to map the underlying vector field of an ODE system to a base vector field, and others [18,19] have attempted to include INNs in Hamiltonian systems.…”

The Symplectic Adjoint Method: Memory-Efficient Backpropagation of Neural-Network-Based Differential Equations

“…Recent developments in automatic differentiation have made it possible to easily train neural networks that have special architectures [29], [30]. Following that, numerous studies have investigated new architectures for specific subclasses of continuous-time dynamical systems, including Hamiltonian systems on canonical coordinates [31], on arbitrary coordinates [32], and with constraints [33], [34], [35], Euler-Lagrange systems [36], and port-Hamiltonian systems [37]. These systems are associated with special geometric structures (e.g., symplectic structure), and the neural networks will be able to learn these systems with high accuracy if their architectures represent such structures well.…”

“…The proposed method obtains the gradient from each step as a numerical integration and is, thus, more robust to rounding errors. 4) Compatible with any ODE systems-while this advantage applies to comparison methods, we emphasize that the proposed method is compatible with any systems described by ODEs [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [43].…”