In this paper we consider a measure-theoretical formulation of the training of NeurODEs in the form of a mean-field optimal control with L 2 -regularization of the control. We derive first order optimality conditions for the NeurODE training problem in the form of a mean-field maximum principle, and show that it admits a unique control solution, which is Lipschitz continuous in time. As a consequence of this uniqueness property, the mean-field maximum principle also provides a strong quantitative generalization error for finite sample approximations. Our derivation of the mean-field maximum principle is much simpler than the ones currently available in the literature for mean-field optimal control problems, and is based on a generalized Lagrange multiplier theorem on convex sets of spaces of measures. The latter is also new, and can be considered as a result of independent interest.

No abstract

Recently a continuous description of the particle swarm optimization (PSO) based on a system of stochastic differential equations was proposed by Grassi and Pareschi in [25] where the authors formally showed the link between PSO and the consensus based optimization (CBO) through zero-inertia limit. This paper is devoted to solving this theoretical open problem proposed in [25] by providing a rigorous derivation of CBO from PSO through the limit of zero inertia, and a quantified convergence rate is obtained as well. The proofs are based on a probabilistic approach by investigating the weak convergence of the corresponding stochastic differential equations (SDEs) of Mckean type in the continuous path space and the results are illustrated with some numerical examples.

No abstract