An evolutionary algorithm that constructs recurrent neural networks

Abstract

Standard methods for inducing both the structure and weight values of recurrent neural networks fit an assumed class of architectures to every task. This simplification is necessary because the interactions between network structure and function are not well understood. Evolutionary computation, which includes genetic algorithms and evolutionary programming, is a population-based search method that has shown promise in such complex tasks. This paper argues that genetic algorithms are inappropriate for network acquisition and describes an evolutionary program, called GNARL, that simultaneously acquires both the structure and weights for recurrent networks. This algorithm's empirical acquisition method allows for the emergence of complex behaviors and topologies that are potentially excluded by the artificial architectural constraints imposed in standard network induction methods. To Appear in: IEEE Transactions on Neural NetworksThe Ohio State University AbstractStandard methods for inducing both the structure and weight values of recurrent neural networks fit an assumed class of architectures to every task. This simplification is necessary because the interactions between network structure and function are not well understood. Evolutionary computation, which includes genetic algorithms and evolutionary programming, is a population-based search method that has shown promise in such complex tasks. This paper argues that genetic algorithms are inappropriate for network acquisition and describes an evolutionary program, called GNARL, that simultaneously acquires both the structure and weights for recurrent networks. This algorithm's empirical acquisition method allows for the emergence of complex behaviors and topologies that are potentially excluded by the artificial architectural constraints imposed in standard network induction methods.

Evolutionary artificial neural networks by multi-dimensional particle swarm optimization

“…assuming a large network, they prune off the superfluous components. However, Angeline et al (1994) indicates that ''Such structural hill climbing methods are susceptible to becoming trapped at structural local minima''. The reasoning behind this is clarified by Miller, Todd, and Hegde (1989), stating that the architecture space is non-differentiable, complex, deceptive and multi-modal.…”

“…When used for training ANNs (with fixed architecture), many researchers (Bartlett & Downs, 1990;Hansen & Meservy, 1996;Miller et al, 1989;Prados, 1992;Porto, Fogel, & Fogel, 1995) reported that GAs can outperform BP in terms of both accuracy and speed, especially for large networks. However, as stated by Angeline et al (1994) and Yao and Liu (1997), GAs do not suit well for evolving networks. For instance, the evolution process by GA suffers from the permutation problem (Belew, McInerney, & Schraudolph, 1991), indicating that two identical ANNs may have different representations.…”

“…EP-based ENNs in Angeline et al (1994) and Yao and Liu (1997) have been proposed to address the aforementioned problems of GAs and many others. The main distinction of EPs from GAs is that they do not use the problematic crossover operation, instead make the commitment to mutation as the sole operator for searching over the weight and architecture spaces.…”

“…Up-to-date designing a (near) optimal network architecture is made by a human expert and requires a tedious trial and error process. Especially automatic determination fall into two broad categories: constructive and pruning algorithms (Burgess, 1994;Frean, 1990;LeCun, Denker, & Solla, 1990;Reed, 1993), where many deficiencies and limitations have been reported (Angeline, Sauders, & Pollack, 1994). Efforts have been then focused on evolutionary algorithms (EAs) (Back & Schwefel, 1993) particularly over Genetic Algorithm (GA) (Goldberg, 1989) and Evolutionary Programming (EP), (Fayyad, Shapire, Smyth, & Uthurusamy, 1996) for both training and evolving ANNs.…”

“…Most GA based methods have also been found quite inadequate for evolving ANNs mainly due to two major problems: permutation problem and noisy fitness evaluation (Yao & Liu, 1997). Although EP-based methods (Angeline et al, 1994;Yao & Liu, 1997;Yao, 1999), might address such problems, they usually suffer from their high complexity and strict parameter dependence.…”