Faster Convergent Artificial Neural Networks

Abstract

Proposed in this paper is a novel fast-convergence algorithm applied to neural networks (ANNs) with a learning rate based on the eigenvalues of the associated Hessian matrix of the input data. That is, the learning rate applied to the backpropagation algorithm changes dynamically with the input data used for training. The best choice of learning rate to converge to an accurate value quickly is derived. This newly proposed fast-convergence algorithm is applied to a traditional multilayer ANN architecture with feed-forward and backpropagation techniques. The proposed strategy is applied to various functions learned by the ANN through training. Learning curves obtained using calculated learning rates according to the novel method proposed are compared to learning curves utilizing an arbitrary learning rate to demonstrate the usefulness of this novel technique. This study shows that convergence to accurate values can be achieved much more quickly (a reduction in iterations by a factor of hundred) using the techniques proposed here. This approach is illustrated in this research work with derivations and pertinent examples to illustrate the method and learning curves obtained.

Optimal Electric-Power Distribution and Load-Sharing on Smart-Grids: Analysis by Artificial Neural Network

“…The final weight matrix [Wij]tr obtained at the convergence of the 20 th training set is then stored. Relevant weight matrix corresponds to a fast-convergence performance of the test ANN as described in [6].…”

“…Further, it is observed that the convergence rate can be speeded up by using the learning rate ( = max) derived on the basis of Hessian matrix applied to the input data as in [6]. It improves the ANN convergence…”

“…As mentioned above, a gradient-based method is employed to adjust the weights proportional to the magnitude of the error, the sign of the error, and the learning rate according to the following equation: De Groff and Neelakanta [6] have shown that the learning rate can be set as inversely proportional to the maximum eigenvalue of the input data set. Convergence depends on the gradient and curvature of the squared error versus the weights.…”

“…Such a larger learning rate may result in the training not converging or diverging. Thus, for applications when the learning rate is too large, to avoid divergence; instead, by setting the learning rate = max , it mostly leads to an optimal solution as indicated in [6].This strategy is implemented in this work.…”

"Forecasting and Surveillance of Covid-19 Pandemic: A Case Study Using Artificial Neural Network "

“…However, since it applies the steepest descent method to update the weights, it suffers from a slow convergence rate and may yield suboptimal solutions [3]. Therefore, in this work a procedure is used that increases the rate of convergence [4]. Applying this faster technique results in the number of iterations required to train the net being much smaller.…”