On the existence of minimizers in shallow residual ReLU neural network optimization landscapes

Abstract

Many mathematical convergence results for gradient descent (GD) based algorithms employ the assumption that the GD process is (almost surely) bounded and, also in concrete numerical simulations, divergence of the GD process may slow down, or even completely rule out, convergence of the error function. In practical relevant learning problems, it thus seems to be advisable to design the ANN architectures in a way so that GD optimization processes remain bounded. The property of the boundedness of GD processes for a given learning problem seems, however, to be closely related to the existence of minimizers in the optimization landscape and, in particular, GD trajectories may escape to infinity if the infimum of the error function (objective function) is not attained in the optimization landscape. This naturally raises the question of the existence of minimizers in the optimization landscape and, in the situation of shallow residual ANNs with multi-dimensional input layers and multi-dimensional hidden layers with the ReLU activation, the main result of this work answers this question affirmatively for a general class of loss functions and all continuous target functions. In our proof of this statement, we propose a kind of closure of the search space, where the limits are called generalized responses, and, thereafter, we provide sufficient criteria for the loss function and the underlying probability distribution which ensure that all additional artificial generalized responses are suboptimal which finally allows us to conclude the existence of minimizers in the optimization landscape.

On minimal representations of shallow ReLU networks

“…We distinguish cases. If one has m 0 = 1 (case (a)) or a ≡ b ∈ R (case (c)), then the proof of Proposition 3.3 in [DJK23] shows that an appropriate replacement of a summand of multiplicity two by two summands of multiplicity one reduces the error which shows (9).…”

“…We will work with more intuitive geometric descriptions of realization functions of networks W ∈ W d as introduced in [DJK23]. We call a network W ∈ W d non-degenerate iff for all j = 1, .…”

“…This will entail Theorem 1.2. For a discussion on sufficient and necessary assumptions on the loss function L that implies err L d < err L d−1 see Proposition 3.5 and Example 3.7 in [DJK23].…”

“…However, for general measures µ not necessarily consisting of a finite number of Dirac measures, the literature on the existence of global minima is very limited. There exist positive results for the approximation of functions in the space L p ([0, 1] d , • p ) with shallow feedforward ANNs using heavyside activation [KKV03], the approximation of one-dimensional Lipschitz continuous target functions with shallow feedforward ANNs using ReLU activation and the standard mean square error [JR22], and the approximation of multi-dimensional, continuous target functions with shallow residual ANNs using ReLU activation, see [DJK23]. Conversely, for several common (smooth) activations such as the standard logistic activation, softplus, arctan, hyperbolic tangent and softsign there, generally, do not exist minimizers in the optimization landscape for smooth target functions (or even polynomials), see [PRV21] and [GJL22].…”

“…Second, we show that the additional discontinuous responses perform worse than representable ones under mild conditions on the optimization problem. Compared to [DJK23], where residual networks are treated, the situation is more complex in classical feedforward ReLU network as treated here. This is caused by the more sophisticated structure of the extended search space, see Definition 2.1.…”