Best k-Layer Neural Network Approximations

Abstract

We show that the empirical risk minimization (ERM) problem for neural networks has no solution in general. Given a training set s 1 , . . . , s n ∈ R p with corresponding responses t 1 , . . . , t n ∈ R q , fitting a k-layer neural network ν θ : R p → R q involves estimation of the weights θ ∈ R m via an ERM:We show that even for k = 2, this infimum is not attainable in general for common activations like ReLU, hyperbolic tangent, and sigmoid functions. In addition, we deduce that if one attempts to minimize such a loss function in the event when its infimum is not attainable, it necessarily results in values of θ diverging to ±∞. We will show that for smooth activations σ(x) = 1/ 1 + exp(−x) and σ(x) = tanh(x), such failure to attain an infimum can happen on a positive-measured subset of responses. For the ReLU activation σ(x) = max(0, x), we completely classify cases where the ERM for a best two-layer neural network approximation attains its infimum. In recent applications of neural networks, where overfitting is commonplace, the failure to attain an infimum is avoided by ensuring that the system of equations t i = ν θ (s i ), i = 1, . . . , n, has a solution. For a two-layer ReLU-activated network, we will

On the Existence of Optimal Shallow Feedforward Networks with ReLU Activation

“…[12,Lemma 27.3]) such that there exists a network configuration achieving zero error and, thus, a global minimum in the search space. For shallow feedforward ANNs using ReLU activation it has been shown that also in the underparametrized regime there exists a global minimum if the ANN has a one-dimensional output [18], whereas there are pathological counterexamples in higher dimensions [19]. However, for general measures µ not necessarily consisting of a finite number of Dirac measures, the literature on the existence of global minima is very limited.…”

“…This phenomenon can also be observed in empirical risk minimization for the hyperbolic tangent activation. As shown in [19], in the underparametrized setting, there exist input data such that for all output data from a set of positive Lebesgue measure there does not exist minimizers in the optimization landscape.…”

On minimal representations of shallow ReLU networks

“…This phenomenon can also be observed in empirical risk minimization for the hyperbolic tangent activation. As shown in [LMQ22], in the underparametrized setting, there exist input data such that for all output data from a set of positive Lebesgue measure there does not exist minimizers in the optimization landscape. It remains an open problem whether for ReLU activation this phenomenon prevails.…”