Neural ODE control for classification, approximation and transport

Abstract

We analyze Neural Ordinary Differential Equations (NODEs) from a control theoretical perspective to address some of the main properties and paradigms of Deep Learning (DL), in particular, data classification and universal approximation. These objectives are tackled and achieved from the perspective of the simultaneous control of systems of NODEs. For instance, in the context of classification, each item to be classified corresponds to a different initial datum for the control problem of the NODE, to be classified, all of them by the same common control, to the location (a subdomain of the euclidean space) associated to each label. Our proofs are genuinely nonlinear and constructive, allowing us to estimate the complexity of the control strategies we develop. The nonlinear nature of the activation functions governing the dynamics of NODEs under consideration plays a key role in our proofs, since it allows deforming half of the phase space while the other half remains invariant, a property that classical models in mechanics do not fulfill. This very property allows to build elementary controls inducing specific dynamics and transformations whose concatenation, along with properly chosen hyperplanes, allows achieving our goals in finitely many steps. The nonlinearity of the dynamics is assumed to be Lipschitz. Therefore, our results apply also in the particular case of the ReLU activation function. We also present the counterparts in the context of the control of neural transport equations, establishing a link between optimal transport and deep neural networks.

A Parabolic Approach to the Control of Opinion Spreading

“…Here and in the sequel ⟨•, •⟩ stands for the euclidean scalar product. Motivated by normalising flows, this result constitutes an L 1 -version of the earlier control result in the Wasserstein distance in [23]. The proof relies on a substantial further development of the methods presented in [23], inspired on the simultaneous or ensemble control of Residual Neural Networks (ResNets) and the corresponding ODE counterparts, the so-called Neural ODEs (nODE), (1.2) x(t) ′ = w(t)σ(⟨a(t), x(t)⟩ + b(t)).…”

“…The techniques we develop, inspired on [23], rely on the fact that nODEs enjoy the property of simultaneous or ensemble control (see also [1,8,17,22,[24][25][26] and [5,9,10,18,25,30] for the controllability and optimal control/generalization frameworks, respectively ).…”

Turnpike in optimal control of PDEs, ResNets, and beyond

“…We refer to several recent works for various different techniques -for instance, [4] and [40], where the authors make use of geometric, Lie bracket techniques for dynamics such as (13.5) and (13.3), for which such tools are quite natural (see [39] for a detailed presentation on Lie algebra techniques for nonlinear control). For more compound neural ODE dynamics, such as (13.6), we refer to [158] and [120], where the controls are built explicitly in a constructive way. In particular, in these works (see also [156]), the link with the closely related topic of universal approximation (see the seminal works [41,143], and also the recent survey [46]) is clearly established.…”

“…A precise characterization of (13.10) in terms of the "complexity" of the data (or even the number of samples 37 n) is not known to our knowledge in this nonlinear setting. Partial results are provided in [158], where a characterization in terms of the fractal dimension of the dataset is provided, but solely when referring to explicitly constructed controls/parameters which interpolate the data, and not those of minimal norm.…”

Optimal actuator design via Brunovsky's normal form

“…Considering systems of the form (5.1) is particularly important in the context of deep learning via continuous-time residual neural networks (ResNets) (see [Weinan, 2017;Esteve et al, 2020;Ruiz-Balet and Zuazua, 2021;Geshkovski, 2021]), which are systems taking the form x (t) = w(t)σ(x(t)) + b(t) in (0, T ).…”