An efficient constrained learning algorithm with momentum acceleration

Abstract

Abstract--An algorithm for efficient learning in feedforward networks is presented. Momentum acceleration isachieved by solving a constrained optimization problem using nonlinear programming techniques. In particular, minimization of the usual mean square error cost function is attempted under an additional condition for which the purpose is to optimize the alignment of the weight update vectors in successive epochs. The algorithm is applied to several benchmark training tasks (exclusive-or, encoder, multiplexer, and counter problems). Its performance, in terms of learning speed and scalability properties, is evaluated and found superior to the performance of reputedly fast variants of the back-propagation algorithm in the above benchmarks.

A new constrained learning algorithm for function approximation by encoding a priori information into feedforward neural networks

“…In order to improve the generalization performance and convergent rate, Perantonis et al [19] proposed incorporating a priori information implicit in the issues into the BP algorithm, which facilitates the learning process leading to better solutions. However, the learning algorithm is complicated and the computational requirement is actually large.…”

A Constructive Approach for Finding Arbitrary Roots of Polynomials by Neural Networks

“…Neural networks, especially multilayer perceptron network (MLPN), have been used successfully in many fields [14]- [24]. Some examples include using two-layered perceptron network to do the factorization of polynomials with two or multiple variables [16], [17], applying one-layered linear perceptron for the inversion of nonsingular matrices [21] and solving linear equations by neural networks [22], [23], etc.…”

Optical Character Recognition Using Novel Feature Extraction & Neural Network Classification Techniques

“…Moreover, our results show attractive generalization ability properties: Compared with on-line BP, ALECO-2 achieved better recognition rates in 4 out of 6 test sets, including substantial improvements in the barc and btimes test sets; m the remaining two test sets, its recognition accuracy was marginally inferior to that of on-line BP. The good generalization ability of ALECO-2 can probably be attributed to the h c t that the cost function is changed monotonically and gradually [5], without the abrupt jumps sometimes involved in learning algorithms which incorporate heuristics in their fbrmulation (including on-line BP). Note that in the same spirit of constrained learning, it is possible to augment ALECO-2 with weight elimination techniques [lo] which will hopefully further improve its generalization ability without adverse effect on its learning speed.…”

“…The use of momentum is based on the expectation that bigger weight steps can be achieved by filtering out high fiequency variations of the error surface in the weight space. ALECO-2 is based on the idea of obtaining optimal weight steps by optimizing, at each epoch of the learning algorithm, the euclidean distance between the current and previous epoch weights [5]. In this way, improved learning speed is achieved.…”

“…As regards the classification stage, a multilayered Gedforward ANN is implemented, which is trained by a recently proposed constrained learning algorithm [5] utilizing rigorous non-linear programming methods in order to make optimal use of momentum acceleration.…”