A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions

Abstract

In this article we study the stochastic gradient descent (SGD) optimization method in the training of fully connected feedforward artificial neural networks with ReLU activation. The main result of this work proves that the risk of the SGD process converges to zero if the target function under consideration is constant. In the established convergence result the considered artificial neural networks consist of one input layer, one hidden layer, and one output layer (with $$d \in {\mathbb {N}}$$ d ∈ N neurons on the input layer, $$H\in {\mathbb {N}}$$ H ∈ N neurons on the hidden layer, and one neuron on the output layer). The learning rates of the SGD process are assumed to be sufficiently small, and the input data used in the SGD process to train the artificial neural networks is assumed to be independent and identically distributed.

Universality of gradient descent neural network training

“…The previous paper [24] contains comparable results for 1d shallow networks. Similar approximation results for gradient flow trained shallow 1d networks are in [33,31], with slightly different assumptions on the target f , more general probability weighted L 2 loss and an alternative proof technique. Other approximation and optimization guarantees rely on alternative optimizers.…”

“…Due to the over-parametrized regime, these optimization results achieve zero training error in discrete sample norms and are therefore not immediately compatible with the approximation literature. There are relatively few papers [1,21,42,15,24,26,30,23,45] that consider approximation and optimization simultaneously.…”

Approximation Results for Gradient Flow Trained Neural Networks

“…The previous paper [24] contains comparable results for 1d shallow networks. Similar approximation results for gradient flow trained shallow 1d networks are in [30,32], with slightly different assumptions on the target f , more general probability weighted L 2 loss and an alternative proof technique. Other approximation and optimization guarantees rely on alternative optimizers.…”

“…• Gradient descent or gradient flow error bounds in continuous L 2 norms can be found in [30,32], and [17,37] The first set of papers uses more general L 2 (P) losses, weighted by a probability measure P of the training samples. For deep networks, they show that the loss converges to zero if the learning target f is piecewise polynomial and for shallow networks if the target is a increasing function.…”

College student activity attendance management system design based on Internet of Things and deep learning technology

“…The way the validation set is divided may result in a large variance in the validation scores. In this case, the best practice is to use the K fold cross validation method [13] . This method divides the available data into K partitions (K is usually 4 or 5) and instantiates K identical models.…”