This paper is concerned with the problem of optimizing deep neural networks with diverse transfer functions using evolutionary methods. Standard evolutionary (SEDeeNN) and cooperative coevolutionary methods (CoDeeNN) were applied to three different architectures characterized by different constraints on neural diversity. It was found that (1) SEDeeNN (but not CoDeeNN) changes parameters uniformly across all layers, (2) both evolutionary approaches can exhibit good convergence and generalization properties, and (3) increased neural diversity improves both convergence and generalization. In addition to clarifying the feasibility of evolutionary deep neural networks, we suggests a guiding principle for synergizing evolutionary and error gradient based approaches through layerchange analysis.

Abstract.Although there have been a few approaches to achieve the goal of fault tolerance by diversifying redundancy of the individual networks that make up a neural network ensemble, some of which include ensembles of neural networks of different sizes, and ensembles of different models of neural networks such as Radial Basis Function Networks and Multilayer Perceptron, there is yet to be an empirical study on hybrid neural networks that makes use of a diverse set of transfer functions, which we would expect to be able to exhibit diverse network architectures, and thus possibly more diverse error patterns. In this paper, we present an approach that uses transfer function diversity to achieve significant results on ensembles. The results show that even with relatively small networks having 5 hidden nodes, and a relatively small ensemble size of just 10 members, the ensemble is able to get competitive results on the Iris data set. It also capable of obtaining competitive results with 20 ensemble members of relatively small networks on other popular data sets such as the Diabetes, Sonar, Hepatitis, and Australian Credit Card problems. In addition to that, it is shown that these results can be achieved with a simple sorting and selection of the Top N solutions of the population, in contrast to other methods of selecting ensemble members that can be computationally expensive, such as selection of the Pareto-front, or hill climbing methods of selection.