Evolving efficient learning algorithms for binary mappings

Abstract

Abstract-Gradient descent training of sigmoidal feed-forward neural networks on binary mappings often gets stuck with some outputs totally wrong. This is because a sum-squared-error cost function leads to weight updates that depend on the derivative of the output sigmoid which goes to zero as the output approaches maximal error. Although it is easy to understand the cause, the best remedy is not so obvious. Common solutions involve modifying the training data, deviating from true gradient descent, or changing the cost function. In general, finding the best learning procedures for particular classes of problem is difficult because each usually depends on a number of interacting parameters that need to be set to optimal values for a fair comparison. In this paper I shall use simulated evolution to optimise all the relevant parameters, and come to a clear conclusion concerning the most efficient approach for learning binary mappings.

Using evolution to improve neural network learning: pitfalls and solutions

“…The neural network architecture and training data were as described above, and a few simple tests determined that 20 hidden units were adequate to learn the training data, and that 1200 data samples per simulated year provided adequate performance estimates for both testing and training purposes. Similar studies [12,13,15] have found empirically that the evolved parameters range over many orders of magnitude, so it was computationally efficient to apply the crossovers and mutations to the logarithms of the parameters, rather than the parameters themselves. The additive mutations were based on a mixture of two sufficiently wide…”

“…Earlier evolutionary studies [13,15,16] found that the evolutionary efficiency depends rather strongly on the initial conditions, i.e. on the distribution of innate parameters across the initial population.…”

“…Consequently, to ensure that the networks can learn at their full potential, each has four learning rates η L , a momentum parameter α, and initial weights w ij (0) generated randomly from the four uniform ranges [-r L , +r L ]. 4 For classification problems, it is appropriate to use the cross-entropy cost function [1,13], and add a standard weight decay regularization term to prevent over-fitting [1], so…”

“…It is also necessary to use simplified versions of genetic encoding and recombination. A number of previous studies [13,15,16,17] have shown that it is sufficient to use direct real valued coding of each evolvable parameter, and a simple recombination approach whereby each child inherits characteristics from its two parents such that each parameter is chosen at random somewhere between the values of its parents, with sufficient added noise (or mutation) to give a reasonable possibility of the parameter falling outside the range spanned by the parents. In this study, each network will have ten such evolved parameters specified in its genotype, and many learnable weights/biases, as discussed above and summarised in Table 1.…”

“…Most biological systems follow some kind of steady state approach, and such a nature inspired approach has previously proved successful for evolving good neural network learners [13,15,16]. A natural formal specification is: The concept of "simulated year" is crucial in this study to allow the learning and evolutionary time-scales to be measured in the same units, which in turn is necessary to set up fair comparisons between the various evolutionary approaches.…”