Evolutionary optimization of neural networks with heterogeneous computation: study and implementation

Abstract

In the optimization of Artificial Neural Networks (ANNs) via Evolutionary Algorithms (EAs) and the implementation of the necessary training for the objective function, there is often a trade-off between efficiency and flexibility. Pure software solutions on general-purpose processors tend to be slow because they do not take advantage of the inherent parallelism, whereas hardware realizations usually rely on optimizations that reduce the range of applicable network topologies, or they attempt to increase processing efficiency by means of low-precision data representation. This paper presents, first of all, a study that shows the need of heterogeneous platform (CPU-GPU-FPGA) to accelerate the optimization of ANNs using genetic algorithms and, secondly, an implementation of a platform based on embedded systems with hardware accelerators implemented in FPGA (Field Pro-grammable Gate Array). The implementation of the individuals on a remote low-cost Altera FPGA allowed us to obtain a 3x-4x acceleration compared with a 2.83 GHz Intel Xeon Quad-Core and 6x-7x compared with a 2.2GHz AMD Opteron Quad-Core 2354.

Optimization of Deep Neural Networks Using SoCs with OpenCL

“… Optimization of the neural network topology via heterogeneous evolutionary computation (this experimentation platform will be implemented and evaluated in this article 4.2 ). Optimization of the initial neural network weights via evolutionary computation (this experimentation platform was implemented and evaluated in [ 20 ]). Final training of a neural network with the topology obtained in Phase 3 and the initial weights obtained in Phase 4.…”

“…Optimization of the initial neural network weights via evolutionary computation (this experimentation platform was implemented and evaluated in [ 20 ]).…”

“… In Phases 3 and 4, many MLP training runs were necessary. In our experiments developed for Phase 4 in [ 20 ], we used the resilient backpropagation algorithm. Now, in the essays proposed for Phase 3 that are presented in Section 4.2 , the RMSpropalgorithm was selected for this purpose based on two considerations: the speed of the algorithm and, more importantly, the ease of hardware implementation for all technologies used in the heterogeneous platform.…”

“…Our main modification to this platform was to add the ability for the host processor to connect (via simple socket clients) to a set of embedded processors (soft or hard) using FPGA technology, thereby providing us with many different system-on-a-chip (SoC) solutions ( Figure 2 ). This modification enabled us to adopt the following alternative implementations of the individuals for the main genetic algorithm: local individuals CPU cores GPU processors remote individuals on FPGAs FPGA0 soft embedded processor FPGA1 soft embedded processor + soft embedded processor with neural instructions FPGA2 soft embedded processor + programmable coprocessor in software (technological solution provided in [ 20 ]) FPGA3 soft embedded processor + reconfigurable coprocessor remote individuals on FPGA-SoCs FPGA-SoC0 hard embedded processor FPGA-SoC1 hard embedded processor + soft embedded processor with neural instructions FPGA-SoC2 hard embedded processor + programmable coprocessor in software FPGA-SoC3 hard embedded processor + reconfigurable coprocessor (technological solution provided in this paper work) remote individuals with OpenCL solutions FPGA-SoC4 hard embedded processor + reconfigurable kernel FPGA4 soft embedded processor + reconfigurable kernel CPU1 + PCI Express GPU kernel CPU2 + PCI Express FPGA kernel …”

“…FPGA2 soft embedded processor + programmable coprocessor in software (technological solution provided in [ 20 ])…”