Accelerating Extreme Search of Multidimensional Functions Based on Natural Gradient Descent with Dirichlet Distributions

Abdulkadirov, Ruslan; Lyakhov, Pavel; Nagornov, Nikolay

doi:10.20944/preprints202209.0124.v1

“…The natural gradient is a product between the Fisher information matrix of certain probability distribution and gradient. Proper probability distributions for natural gradient can be Gaussian, beta-in [8] and Dirichlet in [9]. All noted above algorithms are based on ordinary gradient, which does not give the required accuracy in image recognition problem.…”

Section: Introductionmentioning

confidence: 99%

Improving Image Recognition Accuracy in Neural Networks Using Fractional Natural Gradient Descent

Abdulkadirov¹

2023

Preprint

0

View full text Add to dashboard Cite

<p>This paper proposes a modified natural gradient descent, containing fractional derivatives of Riemann-Liouville, Caputo and Grunwald-Letnikov types. Such approach belongs to information-geometric optimization methods, which take into account not only directions of gradients or momentum parameters, but the convexity (curvature) of minimizing function. This technique, comparing with second order optimization algorithms, lets to increase the rate of convergence. With fractional order derivatives it is possible to adjust the descent toward the neighborhood of the global minimum. In experiments, we demonstrated the increasing accuracy of image recognition on MNIST and CIFAR10, using the proposed optimization algorithm.</p>

show abstract

“…The natural gradient is a product between the Fisher information matrix of certain probability distribution and gradient. Proper probability distributions for natural gradient can be Gaussian, beta-in [8] and Dirichlet in [9]. All noted above algorithms are based on ordinary gradient, which does not give the required accuracy in image recognition problem.…”

Section: Introductionmentioning

confidence: 99%

Improving Image Recognition Accuracy in Neural Networks Using Fractional Natural Gradient Descent

Abdulkadirov¹

2023

Preprint

0

View full text Add to dashboard Cite

<p>This paper proposes a modified natural gradient descent, containing fractional derivatives of Riemann-Liouville, Caputo and Grunwald-Letnikov types. Such approach belongs to information-geometric optimization methods, which take into account not only directions of gradients or momentum parameters, but the convexity (curvature) of minimizing function. This technique, comparing with second order optimization algorithms, lets to increase the rate of convergence. With fractional order derivatives it is possible to adjust the descent toward the neighborhood of the global minimum. In experiments, we demonstrated the increasing accuracy of image recognition on MNIST and CIFAR10, using the proposed optimization algorithm.</p>

show abstract

Survey of Optimization Algorithms in Modern Neural Networks

Abdulkadirov

¹

,

Lyakhov

²

,

Nagornov

³

2023

Self Cite

View full text Add to dashboard Cite

The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced humans in many problems, remains the most well-utilized branch of machine learning. Thus, one must select appropriate neural network architectures, data processing, and advanced applied mathematics tools. A common challenge for these networks is achieving the highest accuracy in a short time. This problem is solved by modifying networks and improving data pre-processing, where accuracy increases along with training time. Bt using optimization methods, one can improve the accuracy without increasing the time. In this review, we consider all existing optimization algorithms that meet in neural networks. We present modifications of optimization algorithms of the first, second, and information-geometric order, which are related to information geometry for Fisher–Rao and Bregman metrics. These optimizers have significantly influenced the development of neural networks through geometric and probabilistic tools. We present applications of all the given optimization algorithms, considering the types of neural networks. After that, we show ways to develop optimization algorithms in further research using modern neural networks. Fractional order, bilevel, and gradient-free optimizers can replace classical gradient-based optimizers. Such approaches are induced in graph, spiking, complex-valued, quantum, and wavelet neural networks. Besides pattern recognition, time series prediction, and object detection, there are many other applications in machine learning: quantum computations, partial differential, and integrodifferential equations, and stochastic processes.

show abstract

Accelerating Extreme Search of Multidimensional Functions Based on Natural Gradient Descent with Dirichlet Distributions

Cited by 3 publications

References 10 publications

Improving Image Recognition Accuracy in Neural Networks Using Fractional Natural Gradient Descent

Improving Image Recognition Accuracy in Neural Networks Using Fractional Natural Gradient Descent

Improving Image Recognition Accuracy in Neural Networks Using Fractional Natural Gradient Descent

Survey of Optimization Algorithms in Modern Neural Networks

Contact Info

Product

Resources

About