A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions

Cheridito, Patrick; Jentzen, Arnulf; Riekert, Adrian; Rossmannek, Florian

doi:10.1016/j.jco.2022.101646

Cited by 16 publications

(10 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, in the experiment, we did not enhance data for initialization. Second, we chose four EfficientNet models, considering that EfficientNet has been proved in numerous literature reviews to have a greater advantage in image deep learning, compared with Densen-Net, ResNet [18], EfficientNetB3, EfficientNetB5, EfficientNetB6, and EfficientNetB7 [40]. EfficientNetB0 to EfficientNetB7 included seven blocks.…”

Section: Datasets and Experimental Settingsmentioning

confidence: 99%

“…Swish has an advantage in big data and deeper complex networks and thus performs more efficiently than others [17]. Proposed by Misra et al, Mish is also a nonmonotonic differentiable function, which has a lower bound but no upper bound [18]. Mish is almost smooth at any point on the curve, allowing the transformation of more valid information into the model to improve accuracy and generalization performance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Smish: A Novel Activation Function for Deep Learning Methods

2022

View full text Add to dashboard Cite

Activation functions are crucial in deep learning networks, given that the nonlinear ability of activation functions endows deep neural networks with real artificial intelligence. Nonlinear nonmonotonic activation functions, such as rectified linear units, Tan hyperbolic (tanh), Sigmoid, Swish, Mish, and Logish, perform well in deep learning models; however, only a few of them are widely used in mostly all applications due to their existing inconsistencies. Inspired by the MB-C-BSIF method, this study proposes Smish, a novel nonlinear activation function, expressed as f(x)=x·tanh[ln(1+sigmoid(x))], which could overcome other activation functions with good properties. Logarithmic operations are first used to reduce the range of sigmoid(x). The value is then calculated using the tanh operator. Inputs are ultimately used to multiply the previous value, thus exhibiting negative output regularization. Experiments show that Smish tends to operate more efficiently than Logish, Mish, and other activation functions on EfficientNet models with open datasets. Moreover, we evaluated the performance of Smish in various deep learning models and the parameters of its function f(x)=αx·tanh[ln(1+sigmoid(βx))], and where α = 1 and β = 1, Smish was found to exhibit the highest accuracy. The experimental results show that with Smish, the EfficientNetB3 network exhibits a Top-1 accuracy of 84.1% on the CIFAR-10 dataset; the EfficientNetB5 network has a Top-1 accuracy of 99.89% on the MNIST dataset; and the EfficientnetB7 network has a Top-1 accuracy of 91.14% on the SVHN dataset. These values are superior to those obtained using other state-of-the-art activation functions, which shows that Smish is more suitable for complex deep learning models.

show abstract

Section: Datasets and Experimental Settingsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Smish: A Novel Activation Function for Deep Learning Methods

2022

View full text Add to dashboard Cite

show abstract

“…Moreover, non-global local minimum points could be found in the risk landscape of ANNs with one hidden layer and ReLU activation in special student-teacher setups with the probability distribution of the input data given by the normal distribution (see Safran & Shamir [31]). In other cases, where the target function has a very simple form, the critical points of the risk landscape are fully characterized and thus all local minimum points are known (see Cheridito et al [2,Corollary 2.15], Cheridito et al [3], and Jentzen & Riekert [17,Corollary 2.11]). Additionally, in the case of ANNs with linear activation and finitely many training data it was shown that all local minimum points of the risk function corresponding to the squared error loss are global minimum points (cf.…”

Section: Introductionmentioning

confidence: 99%

“…To describe a GF trajectory, we need to specify an appropriate generalized gradient function in Theorem 1.2 as the risk function is not differentiable in the case of ANNs with ReLU activation (due to the fact that the ReLU activation function R x → max{x, 0} ∈ R fails to be differentiable in the origin). As in [17] (cf., e.g., also Cheridito et al [2]) we accomplish this by means of an approximation procedure in which the ReLU activation function R x → max{x, 0} ∈ R is approximated through appropriate continuously differentiable functions whose derivatives converge pointwise to the left-derivative of the ReLU activation function; see (1.2) in Theorem 1.2. We now present the precise statement of Theorem 1.2.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On the existence of infinitely many realization functions of non-global local minima in the training of artificial neural networks with ReLU activation

Ibragimov¹,

Jentzen²,

Kröger³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Gradient descent (GD) type optimization schemes are the standard instruments to train fully connected feedforward artificial neural networks (ANNs) with rectified linear unit (ReLU) activation and can be considered as temporal discretizations of solutions of gradient flow (GF) differential equations. It has recently been proved that the risk of every bounded GF trajectory converges in the training of ANNs with one hidden layer and ReLU activation to the risk of a critical point, by which we mean a zero point of the corresponding gradient function. Taking this into account it is one of the key research issues in the mathematical convergence analysis of GF trajectories and GD type optimization schemes, respectively, to study sufficient and necessary conditions for critical points of the risk function and, thereby, to obtain an understanding about the appearance of critical points in dependence of the problem parameters such as the target function. In the first main result of this work we prove in the training of ANNs with one hidden layer and ReLU activation that for every ∈ R, ∈ ( , ∞) and every arbitrarily large positive δ ∈ (0, ∞) we have that there exists a Lipschitz continuous target function : [ , ] → R such that for every number H ∈ N ∩ (1, ∞) of neurons on the hidden layer we have that the risk function has uncountably many different realization functions of non-global local minimum points whose risks are strictly larger than the sum of the risk of the global minimum points and the arbitrarily large positive real number δ. In the second main result of this work we show in the training of ANNs with one hidden layer and ReLU activation in the special situation where there is only one neuron on the hidden layer and where the target function is continuous and piecewise polynomial that there exist at most finitely many different realization functions of critical points.

show abstract

Improvement of Artificial Neural Network Algorithm and Its Application in Hydraulic Measurement Technology

Yuansheng

2022

Application of Intelligent Systems in Multi-Modal Information Analytics

View full text Add to dashboard Cite

A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions

Cited by 16 publications

References 10 publications

Smish: A Novel Activation Function for Deep Learning Methods

Smish: A Novel Activation Function for Deep Learning Methods

On the existence of infinitely many realization functions of non-global local minima in the training of artificial neural networks with ReLU activation

Improvement of Artificial Neural Network Algorithm and Its Application in Hydraulic Measurement Technology

Contact Info

Product

Resources

About