Accelerating Extreme Search of Multidimensional Functions Based on Natural Gradient Descent with Dirichlet Distributions

Abstract

The high accuracy attainment, using less complex architectures of neural networks, remains one of the most important problems in machine learning. In many studies, increasing the quality of recognition and prediction is obtained by extending neural networks with usual or special neurons, which significantly increases the time of training. However, engaging an optimization algorithm, which gives us a value of the loss function in the neighborhood of global minimum, can reduce the number of layers and epochs. In this work, we explore the extreme searching of multidimensional functions by proposed natural gradient descent based on Dirichlet and generalized Dirichlet distributions. The natural gradient is based on describing a multidimensional surface with probability distributions, which allows us to reduce the change in the accuracy of gradient and step size. The proposed algorithm is equipped with step-size adaptation, which allows it to obtain higher accuracy, taking a small number of iterations in the process of minimization, compared with the usual gradient descent and adaptive moment estimate. We provide experiments on test functions in four- and three-dimensional spaces, where natural gradient descent proves its ability to converge in the neighborhood of global minimum. Such an approach can find its application in minimizing the loss function in various types of neural networks, such as convolution, recurrent, spiking and quantum networks.

Survey of Optimization Algorithms in Modern Neural Networks

“…Such networks are part of developing quantum information theory and quantum computers, that makes the process of training significantly faster. But in actual researches, vanilla natural gradient descent with Dirichlet distributions is already tested on Rastrigin and Rosebrock functions in [108] and exploited in convolutional, recurrent neural networks in [109]. The important thing in this method is selecting appropriate probability distribution, which can increase the rate of convergence and even accelerate learning process.…”

“…But selecting appropriate probability distribution, like Gauss and Dirichelt, we can reduce the variable θ in Fisher infromation matrix, what makes possible to avoid its calculation in every iteration. Such approach is realized in [108] - [111]. The natural gradient descent, based on Fisher-Rao metrics, can replace second order optimization algorithms, due to their rate of convergence and time consumption.…”

“…Optimizer Application SGD-type CNN [131], RNN [132], GNN [133], SGD PINN [134], SNN [135], CVNN [136], AE [137] AdaGrad CNN [138] AdaDelta CNN [139], RNN [139], SNN [140] RMSProp CNN [141], RNN [142], SNN [ CNN [144], RNN [145] Adam-type CNN [146], RNN [147], SNN [148], Adam PINN [150], GNN [151], CVNN [ Information geometry CNN [168], RNN [109], NGD GNN [169], PINN [170], QNN [170] MD CNN, RNN [172] Note, that CNN, RNN, GNN, PINN, SNN, CVNN, QNN are convolutional, recurrent, graph, physics-informed, spiking, complex-valued and quantum neural networks, respectively. These neural networks proved their ability to solve vast majority of problems, related to recognition, prediction, generation, processing, detecting and so on.…”

The training response law explains how deep neural networks learn

“…As a training algorithm, we use an optimizer, known as SGD [10]. Ideally, this can be regarded as a potential dynamics for each parameter, w,…”

“…We can update the network parameters along the gradient of the loss step by step. The training dynamics are generated through the update steps and are often stochastic depending on the update procedures [10]. Regardless of the stochastic nature, we can derive a deterministic description of the training response at some ensemble levels [11,12,15].…”