As an enormous computing power is required to get knowledge from a large volume of data, the parallel and distributed computing is highly recommended to process them. Artificial Neural Networks (ANNs) need as much as possible data to have high accuracy, whereas parallel processing can help us to save time in ANNs training. In this paper, exemplary parallelization of artificial neural network training by dint of Java and its native socket libraries has been implemented. During the experiments, it has been noticed that Java native socket implementation tends to have memory issues when a large amount of training datasets are involved in training. It has been remarked that exemplary parallelization of artificial neural network training cannot outperform drastically when additional nodes are introduced into the system after a certain point. This is comprehensively due to the network communication complexity in the system.Corresponding Author: Name Md. Haidar Sharif Affiliation International Balkan University Address Skopje, Republic of Macedonia Phone +38923214831 Email msharif@ibu.edu.mk
IntroductionParallel and distributed computing are required to get knowledge from a large volume of data. Artificial Neural Networks (ANNs) need to have high accuracy, whereas parallel processing can help us to save time in ANNs training. In the literature, many articles on parallelization of neural network pieces of training can be found. Some examples are given herewith. A parallelized Back-Propagation Neural Network (BPNN) based on MapReduce computing model was introduced by Liu et al. [1]. A cascading model based classification approach was implemented to introduce fault tolerance, data replication, and load balancing. Nodes were trained for a specific class available in training dataset. As a result, a much stronger classifier could be built by the weak classifiers in the cluster. A parallelizing technique for the training of neural networks on the cluster computer was presented by Dahl et al. [2]. A full neural network was copied at each cluster node for the eight-bit parity problem. The subset of the training data in each node was randomly selected at each epoch, which is the date and time related to a computer's clock [3]. An improvement up to a factor of 11 in terms of training time as compared to sequential training was demonstrated through the obtained results. The type of neural network problem was mentioned. A different kind of exemplary parallelization was implemented. The communication costs of the approach were analysed. But it was not recommended the node parallelism in cluster computer. A comparative study along with an own technique for the performance of exemplary parallel and node parallel strategies on a cluster computer were performed by Pethick et al. [4]. The performance of various sizes of neural networks, miscellaneous dataset sizes, and the number of processors was studied. The affirmative and the negative sides of their proposed method were discussed. The cost equation introduced in their method ...