Improving Extreme Search with Natural Gradient Descent Using Dirichlet Distribution

Abdulkadirov, Ruslan; Lyakhov, Pavel

doi:10.1007/978-3-030-97020-8_3

Cited by 5 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We verified the capability of the proposed algorithm to converge in the neighborhood of the global minimum in the case of the Rastrigin and Rosenbrock functions, where known algorithms do not achieve the global minimum. Such experiments differ from the experiments in [14,15].…”

Section: Discussioncontrasting

confidence: 63%

“…In [14], which is a continuation of [15], we explored the natural gradient descent based on Dirichlet distribution. In this research, we added and calculated the Fisher information matrix of the generalized Dirichlet distribution.…”

Section: Discussionmentioning

confidence: 99%

“…where x (0) = x 0 . The Fisher information matrix F(x (k) ) from [11,14,17] can be calculated on the manifold of probability distributions, whose curvature we use to minimize the continuous function f (θ). Presume that p(x; θ) is a family of probability distributions over space variables x with a parametric vector θ ∈ R n .…”

Section: Natural Gradient Descent and K-l Divergencementioning

confidence: 99%

“…We demonstrate that the natural gradient descent with step-size adaptation based on Dirichlet and generalized Dirichlet distributions has higher accuracy and does not take a large number of iterations for minimizing test functions compared to gradient descent and Adam. Such an approach is a continuation of works [14,15], where the final results did not present the ability of natural gradient descent to converge in the neighborhood of the global minimum.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Accelerating Extreme Search of Multidimensional Functions Based on Natural Gradient Descent with Dirichlet Distributions

2022

View full text Add to dashboard Cite

The high accuracy attainment, using less complex architectures of neural networks, remains one of the most important problems in machine learning. In many studies, increasing the quality of recognition and prediction is obtained by extending neural networks with usual or special neurons, which significantly increases the time of training. However, engaging an optimization algorithm, which gives us a value of the loss function in the neighborhood of global minimum, can reduce the number of layers and epochs. In this work, we explore the extreme searching of multidimensional functions by proposed natural gradient descent based on Dirichlet and generalized Dirichlet distributions. The natural gradient is based on describing a multidimensional surface with probability distributions, which allows us to reduce the change in the accuracy of gradient and step size. The proposed algorithm is equipped with step-size adaptation, which allows it to obtain higher accuracy, taking a small number of iterations in the process of minimization, compared with the usual gradient descent and adaptive moment estimate. We provide experiments on test functions in four- and three-dimensional spaces, where natural gradient descent proves its ability to converge in the neighborhood of global minimum. Such an approach can find its application in minimizing the loss function in various types of neural networks, such as convolution, recurrent, spiking and quantum networks.

show abstract

Section: Discussioncontrasting

confidence: 63%

Section: Discussionmentioning

confidence: 99%

Section: Natural Gradient Descent and K-l Divergencementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Accelerating Extreme Search of Multidimensional Functions Based on Natural Gradient Descent with Dirichlet Distributions

2022

View full text Add to dashboard Cite

show abstract

“…But selecting appropriate probability distribution, like Gauss and Dirichelt, we can reduce the variable θ in Fisher infromation matrix, what makes possible to avoid its calculation in every iteration. Such approach is realized in [108] - [111]. The natural gradient descent, based on Fisher-Rao metrics, can replace second order optimization algorithms, due to their rate of convergence and time consumption.…”

Section: Probability Density Functionmentioning

confidence: 99%

Survey of Optimization Algorithms in Modern Neural Networks

Abdulkadirov¹,

Lyakhov²,

Nagornov³

2023

Preprint

View full text Add to dashboard Cite

Creating self-learning algorithms, developing deep neural networks and improving other methods that "learn" for various areas of human activity is the main goal of the theory of machine learning. It helps to replace the human with a machine, aiming to increase the quality of production. The theory of artificial neural networks, which already have replaced the humans in problems of detection of moving objects, recognition of images or sounds, time series prediction, big data analysis and numerical methods remains the most dispersed branch of the theory of machine learning. Certainly, for each area of human activity it is necessary to select appropriate neural network architectures, methods of data processing and some novel tools from applied mathematics. But the universal problem for all these neural networks with specific data is the achieving the highest accuracy in short time. Such problem can be resolved by increasing sizes of architectures and improving data preprocessing, where the accuracy rises with the training time. But there is a possibility to increase the accuracy without time growing, applying optimization methods. In this survey we demonstrate existing optimization algorithms of all types, which can be used in neural networks. There are presented modifications of basic optimization algorithms, such as stochastic gradient descent, adaptive moment estimation, Newton and quasi-Newton optimization methods. But the most recent optimization algorithms are related to information geometry, for Fisher-Rao and Bregman metrics. This approach in optimization extended the theory of classic neural networks to quantum and complex-valued neural networks, due to geometric and probabilistic tools. There are provided applications of all introduced optimization algorithms, what delighted many kinds of neural networks, which can be improved by including any advanced approaches in minimization of the loss function. Afterwards, we demonstrated ways of developing optimization algorithms in further researches, engaging neural networks with progressive architectures. Classical gradient based optimizers can be replaced by fractional order, bilevel and, even, gradient free optimization methods. There is a possibility to add such analogues in graph, spiking, complex-valued, quantum and wavelet neural networks. Besides the usual problems of image recognition, time series prediction, object detection, there are many are other tasks for modern theory of machine learning, such as solving problem of quantum computations, partial differential and integro-differential equations, stochastic processes and Brownian motion, making decisions and computer algebra.

show abstract

Solving Poisson Equation by Physics-Informed Neural Network with Natural Gradient Descent with Momentum

Abdulkadirov,

Lyakhov,

Nagornov

2023

2023 Seminar on Signal Processing

View full text Add to dashboard Cite

Improving Extreme Search with Natural Gradient Descent Using Dirichlet Distribution

Cited by 5 publications

References 7 publications

Accelerating Extreme Search of Multidimensional Functions Based on Natural Gradient Descent with Dirichlet Distributions

Accelerating Extreme Search of Multidimensional Functions Based on Natural Gradient Descent with Dirichlet Distributions

Survey of Optimization Algorithms in Modern Neural Networks

Solving Poisson Equation by Physics-Informed Neural Network with Natural Gradient Descent with Momentum

Contact Info

Product

Resources

About