Stateful ODE-Nets using Basis Function Expansions

Abstract

The recently-introduced class of ordinary differential equation networks (ODE-Nets) establishes a fruitful connection between deep learning and dynamical systems. In this work, we reconsider formulations of the weights as continuous-depth functions using linear combinations of basis functions. This perspective allows us to compress the weights through a change of basis, without retraining, while maintaining near state-of-the-art performance. In turn, both inference time and the memory footprint are reduced, enabling quick and rigorous adaptation between computational environments. Furthermore, our framework enables meaningful continuous-in-time batch normalization layers using function projections. The performance of basis function compression is demonstrated by applying continuous-depth models to (a) image classification tasks using convolutional units and (b) sentence-tagging tasks using transformer encoder units.

Long Expressive Memory for Sequence Modeling

“…The use of ODE-based learning architectures has also received considerable attention in recent years with examples such as continuous-time neural ODEs [9,34,33] and their recurrent extensions ODE-RNNs [35], as well as RNNs based on discretizations of ODEs [7,12,10,28,36,37].…”