Strong Stationarity for Optimal Control Problems with Non-smooth Integral Equation Constraints: Application to a Continuous DNN

Abstract

Motivated by the residual type neural networks (ResNet), this paper studies optimal control problems constrained by a non-smooth integral equation associated to a fractional differential equation. Such non-smooth equations, for instance, arise in the continuous representation of fractional deep neural networks (DNNs). Here the underlying non-differentiable function is the ReLU or max function. The control enters in a nonlinear and multiplicative manner and we additionally impose control constraints. Because of the presence of the non-differentiable mapping, the application of standard adjoint calculus is excluded. We derive strong stationary conditions by relying on the limited differentiability properties of the non-smooth map. While traditional approaches smoothen the non-differentiable function, no such smoothness is retained in our final strong stationarity system. Thus, this work also closes a gap which currently exists in continuous neural networks with ReLU type activation function.

Strong Stationarity for the Control of Viscous History-Dependent Evolutionary Vis Arising in Applications

“…[26,41,15] and the references therein. Regarding strong stationarity for optimal control of nonsmooth evolution processes, the literature is very scarce and the only papers known to the author addressing this issue so far are [13,14,8,12] (EVIs) and [22,8,2] (time-dependent PDEs/ODEs). We point out that, in contrast to our problem, all the above mentioned contributions on the topic of strong stationarity for EVIs do not take a history operator into account.…”