Closed-form continuous-time neural networks

Abstract

Continuous-time neural networks are a class of machine learning systems that can tackle representation learning on spatiotemporal decision-making tasks. These models are typically represented by continuous differential equations. However, their expressive power when they are deployed on computers is bottlenecked by numerical differential equation solvers. This limitation has notably slowed down the scaling and understanding of numerous natural physical phenomena such as the dynamics of nervous systems. Ideally, we would circumvent this bottleneck by solving the given dynamical system in closed form. This is known to be intractable in general. Here, we show that it is possible to closely approximate the interaction between neurons and synapses—the building blocks of natural and artificial neural networks—constructed by liquid time-constant networks efficiently in closed form. To this end, we compute a tightly bounded approximation of the solution of an integral appearing in liquid time-constant dynamics that has had no known closed-form solution so far. This closed-form solution impacts the design of continuous-time and continuous-depth neural models. For instance, since time appears explicitly in closed form, the formulation relaxes the need for complex numerical solvers. Consequently, we obtain models that are between one and five orders of magnitude faster in training and inference compared with differential equation-based counterparts. More importantly, in contrast to ordinary differential equation-based continuous networks, closed-form networks can scale remarkably well compared with other deep learning instances. Lastly, as these models are derived from liquid networks, they show good performance in time-series modelling compared with advanced recurrent neural network models.

Noise Driven Coupled Nonlinear Systems: A “Dynamical Multilayer Perceptron” Demonstrating XOR Functionality

“…The similarity of our system to the newly-introduced "liquid networks" [18] is notable. Analogous to the liquid networks, our noisy system yields a random 2-state time series solution, however the relevant information, for our purposes, is contained in the long-time probability density function P (y 1 ) of the readout element; the relative heights of the peaks of the density function are germane to the realization of the XOR.…”

Efficient Edge-AI Models for Robust ECG Abnormality Detection on Resource-Constrained Hardware

“…A notable characteristic of Closed-form Control (CfC) neural networks is that they do not rely on numerical Ordinary Differential Equation (ODE) solvers to generate their temporal rollouts [9]. This kind of network not only achieves the flexible, causal and continuous-time feature of ODE-based networks but also has a better efficiency compared to them [9]. The CfC model can be represented by the equation 3, where σ (− f ( x, I ; θ f ) t ) and [1 − σ (− [ f ( x, I ; θ f )] t )] are the time-continuous gating [9].…”

“…This kind of network not only achieves the flexible, causal and continuous-time feature of ODE-based networks but also has a better efficiency compared to them [9]. The CfC model can be represented by the equation 3, where σ (− f ( x, I ; θ f ) t ) and [1 − σ (− [ f ( x, I ; θ f )] t )] are the time-continuous gating [9]. …”

Artificial intelligence system for enhanced automated 1,3-propanediol green biosynthesis

“…Recently, machine learning technology has set off a revolution in traditional subjects such as biology. 35 Artificial neural networks (ANN) show advantages in complex modeling problems, including protein structure prediction, 36 neuron simulation, 37 and the discovery of underlying dynamics in biological systems. 38 For the traditional fermentation processes, artifi-cial intelligence (AI) has been considered as a powerful tool for process optimization and control.…”