Apache Spark as a Tool for Parallel Population-Based Optimization

Jędrzejowicz, Piotr; Wierzbowska, Izabela

doi:10.1007/978-981-13-8311-3_16

Cited by 4 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [188], Spark is used for developing a Particle Swarm Optimization and a Differential Evolution algorithm. Finally, authors of [189] introduce a parallel population-based optimization algorithm with Spark. Another interesting work along this line is [190], in which a scalable Genetic Algorithm is developed using Apache Spark.…”

Section: Bio-inspired Computation For Data Processing and Learningmentioning

confidence: 99%

Bio-inspired computation for big data fusion, storage, processing, learning and visualization: state of the art and future directions

Torre-Bastida

Díaz-de-Arcaya

Osaba

et al. 2021

Neural Comput & Applic

View full text Add to dashboard Cite

This overview gravitates on research achievements that have recently emerged from the confluence between Big Data technologies and bio-inspired computation. A manifold of reasons can be identified for the profitable synergy between these two paradigms, all rooted on the adaptability, intelligence and robustness that biologically inspired principles can provide to technologies aimed to manage, retrieve, fuse and process Big Data efficiently. We delve into this research field by first analyzing in depth the existing literature, with a focus on advances reported in the last few years. This prior literature analysis is complemented by an identification of the new trends and open challenges in Big Data that remain unsolved to date, and that can be effectively addressed by bio-inspired algorithms. As a second contribution, this work elaborates on how bio-inspired algorithms need to be adapted for their use in a Big Data context, in which data fusion becomes crucial as a previous step to allow processing and mining several and potentially heterogeneous data sources. This analysis allows exploring and comparing the scope and efficiency of existing approaches across different problems and domains, with the purpose of identifying new potential applications and research niches. Finally, this survey highlights open issues that remain unsolved to date in this research avenue, alongside a prescription of recommendations for future research.

show abstract

Section: Bio-inspired Computation For Data Processing and Learningmentioning

confidence: 99%

Bio-inspired computation for big data fusion, storage, processing, learning and visualization: state of the art and future directions

Torre-Bastida

Díaz-de-Arcaya

Osaba

et al. 2021

Neural Comput & Applic

View full text Add to dashboard Cite

show abstract

“…Spark-based ant colony optimization algorithm for solving the TSP was proposed in [13]. A Sparkbased version of the population learning metaheuristic applied, among others, to job-shop scheduling problem (JSSP) can be found in [14]. Some extensive reviews of developments in the field of parallel metaheuristics can be found in [15], [16], and [17].…”

Section: Introductionmentioning

confidence: 99%

Mushroom Picking Framework with Cache Memories for Solving Job Shop Scheduling Problem

Jędrzejowicz,

Wierzbowska

2023

Annals of Computer Science and Information Systems

View full text Add to dashboard Cite

Applying population-based metaheuristics is a known method of solving difficult optimization problems. In this paper the search for the best solution is conducted by decentralized, self-organized agents, working in parallel threads, in the so called mushroom-picking method. The search is enhanced by remembering in which part of the recently improved solution the last successful change took place and intensifying the search in this part. A computational experiment shows that introducing the component for remembering the most recent changes may improve the results obtained by the model in the case of JSSP problems.

show abstract

“…Distributed ant colony optimization algorithm for solving the TSP was proposed in [25]. Parallel version of the population learning metaheuristic applied, among other, to TSP can be found in [26].…”

Section: Introductionmentioning

confidence: 99%

“…Performance of the so far developed algorithms and methods for solving TSP and JSSP still leaves a room for improvements in terms of both minimization of computation error and computation time; The idea of employing a set of simple asynchronous agents exploring the solution space and trying to improve currently encountered solutions has already proven successful in numerous applications of the authors (see for example [26,38,39]).…”

mentioning

confidence: 99%

Parallelized Swarm Intelligence Approach for Solving TSP and JSSP Problems

Jędrzejowicz

Wierzbowska

2020

Algorithms

Self Cite

View full text Add to dashboard Cite

One of the possible approaches to solving difficult optimization problems is applying population-based metaheuristics. Among such metaheuristics, there is a special class where searching for the best solution is based on the collective behavior of decentralized, self-organized agents. This study proposes an approach in which a swarm of agents tries to improve solutions from the population of solutions. The process is carried out in parallel threads. The proposed algorithm—based on the mushroom-picking metaphor—was implemented using Scala in an Apache Spark environment. An extended computational experiment shows how introducing a combination of simple optimization agents and increasing the number of threads may improve the results obtained by the model in the case of TSP and JSSP problems.

show abstract

Apache Spark as a Tool for Parallel Population-Based Optimization

Cited by 4 publications

References 18 publications

Bio-inspired computation for big data fusion, storage, processing, learning and visualization: state of the art and future directions

Bio-inspired computation for big data fusion, storage, processing, learning and visualization: state of the art and future directions

Mushroom Picking Framework with Cache Memories for Solving Job Shop Scheduling Problem

Parallelized Swarm Intelligence Approach for Solving TSP and JSSP Problems

Contact Info

Product

Resources

About