Implementing Parallel Differential Evolution on Spark

Teijeiro, Diego; Pardo, Xoán C.; González, Patricia; Banga, Julio R.; Doallo, Ramón

doi:10.1007/978-3-319-31153-1_6

Cited by 28 publications

(22 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Research on cloud-oriented parallel metaheuristics based mainly on the use of MapReduce 18 and Spark 19 has also received increasing attention in recent years. [20][21][22][23][24][25][26] The experimental results reveal that the extra cost of the I/O operations and the system bookkeeping overhead significantly reduces the performance of the parallelization using MapReduce in iterative algorithms. This performance can be improved by an order of magnitude when using Spark.…”

Section: Related Workmentioning

confidence: 99%

Multimethod optimization in the cloud: A case‐study in systems biology modelling

González

Penas

Pardo

et al. 2018

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Summary Optimization problems appear in many different applications in science and engineering. A large number of different algorithms have been proposed for solving them; however, there is no unique general optimization method that performs efficiently across a diverse set of problems. Thus, a multimethod optimization, in which different algorithms cooperate to outperform the results obtained by any of them in isolation, is a very appealing alternative. Besides, as real‐life optimization problems are becoming more and more challenging, the use of HPC techniques to implement these algorithms represents an effective strategy to speed up the time‐to‐solution. In addition, a parallel multimethod approach can benefit from the effortless access to q large number of distributed resources facilitated by cloud computing. In this paper, we propose a self‐adaptive cooperative parallel multimethod for global optimization. This proposal aims to perform a thorough exploration of the solution space by means of multiple concurrent executions of a broad range of search strategies. For its evaluation, we consider an extremely challenging case‐study from the field of computational systems biology. We also assess the performance of the proposal on a public cloud, demonstrating both the potential of the multimethod approach and the opportunity that the cloud provides for these problems.

show abstract

Section: Related Workmentioning

confidence: 99%

Multimethod optimization in the cloud: A case‐study in systems biology modelling

González

Penas

Pardo

et al. 2018

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…This algorithm was implemented in three levels, each of which consists of DE operations. The use of Spark for the parallelization of the DE algorithm was explored in [12]. In that paper Sparkbased implementations of two different parallel schemes of the DE algorithm, the master-slave and the island-based, are proposed and evaluated.…”

Section: Background and Related Workmentioning

confidence: 99%

“…The implementation of the DE master-slave model does not fit well with the distributed nature of programming models like MR or Spark [12]. The reason is that when the mutation strategy is applied to each individual, random different individuals have to be selected from the whole population.…”

Section: Algorithm 1: Differential Evolution Algorithmmentioning

confidence: 99%

Evaluation of Parallel Differential Evolution Implementations on MapReduce and Spark

Teijeiro

Pardo

Penas³

et al. 2017

Euro-Par 2016: Parallel Processing Workshops

Self Cite

View full text Add to dashboard Cite

Abstract. Global optimization problems arise in many areas of science and engineering, computational and systems biology and bioinformatics among them. Many research efforts have focused on developing parallel metaheuristics to solve them in reasonable computation times. Recently, new programming models are being proposed to deal with large scale computations on commodity clusters and Cloud resources. In this paper we investigate how parallel metaheuristics deal with these new models by the parallelization of the popular Differential Evolution algorithm using MapReduce and Spark. The performance evaluation has been carried out both in a local cluster and in the Amazon Web Services public cloud. The results obtained can be particularly useful for those interested in the potential of new Cloud programming models for parallel metaheuristic methods in general and Differential Evolution in particular.

show abstract

“…Deng et al [19] proposed a parallel DE based on RDD, which can decrease the computational time of the objective function. Teijeiro et al [20] presented two different parallel schemes based on Spark for DE algorithm. For LSGO problems, Peng et al [21] designed a Spark-based DE with commensal learning and uniform local search.…”

Section: Introductionmentioning

confidence: 99%

A Hybrid Mechanism of Particle Swarm Optimization and Differential Evolution Algorithms based on Spark

2019

KSII TIIS

View full text Add to dashboard Cite

With the onset of the big data age, data is growing exponentially, and the issue of how to optimize large-scale data processing is especially significant. Large-scale global optimization (LSGO) is a research topic with great interest in academia and industry. Spark is a popular cloud computing framework that can cluster large-scale data, and it can effectively support the functions of iterative calculation through resilient distributed datasets (RDD). In this paper, we propose a hybrid mechanism of particle swarm optimization (PSO) and differential evolution (DE) algorithms based on Spark (SparkPSODE). The SparkPSODE algorithm is a parallel algorithm, in which the RDD and island models are employed. The island model is used to divide the global population into several subpopulations, which are applied to reduce the computational time by corresponding to RDD's partitions. To preserve population diversity and avoid premature convergence, the evolutionary strategy of DE is integrated into SparkPSODE. Finally, SparkPSODE is conducted on a set of benchmark problems on LSGO and show that, in comparison with several algorithms, the proposed SparkPSODE algorithm obtains better optimization performance through experimental results.

show abstract

Implementing Parallel Differential Evolution on Spark

Cited by 28 publications

References 20 publications

Multimethod optimization in the cloud: A case‐study in systems biology modelling

Multimethod optimization in the cloud: A case‐study in systems biology modelling

Evaluation of Parallel Differential Evolution Implementations on MapReduce and Spark

A Hybrid Mechanism of Particle Swarm Optimization and Differential Evolution Algorithms based on Spark

Contact Info

Product

Resources

About