This paper is concerned with the problem of optimizing deep neural networks with diverse transfer functions using evolutionary methods. Standard evolutionary (SEDeeNN) and cooperative coevolutionary methods (CoDeeNN) were applied to three different architectures characterized by different constraints on neural diversity. It was found that (1) SEDeeNN (but not CoDeeNN) changes parameters uniformly across all layers, (2) both evolutionary approaches can exhibit good convergence and generalization properties, and (3) increased neural diversity improves both convergence and generalization. In addition to clarifying the feasibility of evolutionary deep neural networks, we suggests a guiding principle for synergizing evolutionary and error gradient based approaches through layerchange analysis.