Neural Network Supervised Training Based on a Dimension Reducing Method

Adaptive Algorithms for Neural Network Supervised Learning: A Deterministic Optimization Approach

“…Le Cun's technique could be used to determine an initial η at an additional cost in the number of presentations of the training set during early training. Another approach proposed in the literature is to consider a learning rate that is proportional to the inverse of the Lipschitz constant which, unfortunately, is not easily available [Armijo, 1966;Magoulas et al, 1997aMagoulas et al, , 1997b. So despite these efforts, obtaining convergence of BP training algorithms utilizing a constant learning rate is still considered very difficult [Kuan & Hornik, 1991;Liu et al, 1995] and practitioners usually chose 0 < η < 1 to ensure that successive weight updates will not lead to missing a minimum of the error surface.…”

“…Next we examine approaches to dynamically adapt the rate of learning that are based on optimization methods Magoulas et al, 1997aMagoulas et al, , 1997bVrahatis et al, 2000a;Anastasiadis et al, 2005a;Anastasiadis et al, 2005b]. In the context of unconstrained optimisation, Armijo's modified SD algorithm automatically adapts the rate of convergence [Armijo, 1966].…”

“…It can be shown, in a deterministic framework of analysis, that the weight update equation (16) converges to the point w * (see [Magoulas et al, 1997a[Magoulas et al, , 1997b). However, in the neural network implementation, neither the morphology of the error surface nor the value of L are known a priori.…”

“…Thus, it provides a damping effect in order to avoid oscillations and keeps the descent direction. This strategy also allows to guarantee the convergence of the algorithm due to Armijo [Magoulas et al, 1997a[Magoulas et al, , 1997b.…”

Supervised Training Using Global Search Methods

“…This equation formulates the error function to be minimized, in which t j p speci es the desired response at the j{th neuron of the output layer at the input pattern p and y L j p is the output at the j{th neuron of the output layer L that depends on the weights of the network and is a nonlinear activation function, such as the well known sigmoid Attempts to speed up back{propagation training have been made by dynamically adapting the stepsize during training 9, 20], or by using second derivative related information 11,13,19]. However, these BP{ like training algorithms occasionally converge to local minima which a ect the e ciency of the learning process.…”