Scaling Evolutionary Programming With the Use of Apache Spark

Abstract

Organizations across the globe gather more and more data, encouraged by easyto-use and cheap cloud storage services. Large datasets require new approaches to analysis and processing, which include methods based on machine learning. In particular, symbolic regression can provide many useful insights. Unfortunately, due to high resource requirements, use of this method for large-scale dataset analysis might be unfeasible. In this paper, we analyze a bottleneck in the open-source implementation of this method we call hubert. We identify that the evaluation of individuals is the most costly operation. As a solution to this problem, we propose a new evaluation service based on the Apache Spark framework, which attempts to speed up computations by executing them in a distributed manner on a cluster of machines. We analyze the performance of the service by comparing the evaluation execution time of a number of samples with the use of both implementations. Finally, we draw conclusions and outline plans for further research.

Continuous self‐adaptation of control policies in automatic cloud management

“…The datasets used, generated, and analyzed during the current study are available in publicly accessible repository 20 or can be provided from the corresponding author on a reasonable request.…”

“…This allows the management system to respond to a potentially changing workload while addressing the issues described above. This article's contribution comprises a novel architecture of an autonomous management system which utilizes a continuous policy improvement loop, initial policy training procedure, implementation of the described concepts available as an open source project, 20 experiments demonstrating the functioning of such a management system (comparison with a static policy, reacting to changes in environment) and analysis of their results.…”

Genetic Programming over Spark for Higgs Boson Classification

“…The reducer aggregates the previously created vectors. Funika et al [7] implement an 'Evaluation Service' that can be solicited through a REST API. It is not an implementation of a specific EA algorithm but rather an outsourcing of the evaluation.…”

“…The evaluation represents more than 80% of the total time cost in EA [7,9]. We suggest to focus on evaluation which can be easily distributed on Spark cluster even with limited resources seeing that we do not need independent populations.…”