Performance-aware scheduling of streaming applications using genetic algorithm

Smirnov, Alexander; Melnik, Mikhail; Nasonov, Denis

doi:10.1016/j.procs.2017.05.249

Cited by 24 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of static strategies are topology-aware, such as those listed in [8][9][10], while others are based on resource handling (resource-aware), such as those in [11,12] or [13]. Finally, other recent works employ the idea of linear programming; for example, this is evident in [14][15][16][17].…”

Section: Related Workmentioning

confidence: 99%

Pipelined Dynamic Scheduling of Big Data Streams

Souravlas

Anastasiadou

2020

Applied Sciences

View full text Add to dashboard Cite

We are currently living in the big data era, in which it has become more necessary than ever to develop “smart” schedulers. It is common knowledge that the default Storm scheduler, as well as a large number of static schemes, has presented certain deficiencies. One of the most important of these deficiencies is the weakness in handling cases in which system changes occur. In such a scenario, some type of re-scheduling is necessary to keep the system working in the most efficient way. In this paper, we present a pipeline-based dynamic modular arithmetic-based scheduler (PMOD scheduler), which can be used to re-schedule the streams distributed among a set of nodes and their tasks, when the system parameters (number of tasks, executors or nodes) change. The PMOD scheduler organizes all the required operations in a pipeline scheme, thus reducing the overall processing time.

show abstract

Section: Related Workmentioning

confidence: 99%

Pipelined Dynamic Scheduling of Big Data Streams

Souravlas

Anastasiadou

2020

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Smirnov et al [26] proposed another topology-aware strategy that also takes into consideration system's performance. Their approach is based on a genetic algorithm (GA) and uses performance models of executors on specified nodes to estimate their throughput.…”

Section: B Heuristic Approachesmentioning

confidence: 99%

“…Linear programming is widely used in task scheduling approaches found in literature (e.g. [14], [17], [25], [26]). In such cases, the allocation problem is considered as a linear programming problem.…”

Section: Dsp Frameworkmentioning

confidence: 99%

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

et al. 2020

View full text Add to dashboard Cite

Nowadays, there is an accelerating need to efficiently and timely handle large amounts of data that arrives continuously. Streams of big data led to the emergence of several Distributed Stream Processing Systems (DSPS) that assign processing tasks to the available resources (dynamically or not) and route streaming data between them. Efficient scheduling of processing tasks can reduce application latencies and eliminate network congestions. However, the available DSPSs' in-built scheduling techniques are far from optimal. In this work, we extend our previous work [1], where we proposed a linear scheme for the task allocation and scheduling problem. Our scheme takes advantage of pipelines to handle efficiently applications, where there is need for heavy communication (all-to-all) between tasks assigned to pairs of components. In this work, we prove that our scheme is periodic, we provide a communication refinement algorithm and a mechanism to handle many-to-one assignments efficiently. For concreteness, our work is illustrated based on Apache Storm semantics. The performance evaluation depicts that our algorithm achieves load balance and constraints the required buffer space. For throughput testing, we compared our work to the default Storm scheduler, as well as to R-Storm. Our scheme was found to outperform both the other strategies and achieved an average of 25%-40% improvement compared to Storm's default scheduler under different scenarios, mainly as a result of reduced buffering (≈ 45% less memory). Compared to Rstorm, the results indicate an average of 35%-45% improvement.

show abstract

“…Each gen in chromosome represents quantity of the product. On each chromosome, fitness values are calculated to determine the quality of the solutions [24]. Fitness function of product storage optimization has shown in (6).…”

Section: Hybrid Rcga-vnsmentioning

confidence: 99%

Hybrid Real-Coded Genetic Algorithm and Variable Neighborhood Search for Optimization of Product Storage

Rikatsih

Mahmudy

Syafrial

2019

JITeCS

View full text Add to dashboard Cite

Agricultural product storage has a problem that need to be noticedbecause it has an impact in gaining the profit according to the number ofproducts and the capacity of storage. Inappropriate combination of productcauses high expenses and low profit. To solve the problem, we propose geneticalgorithm (GA) as the optimization method. Although GA is good enough tosolve the problem, GA not always gives an optimum result in complex searchspaces because it is easy to be trapped in local optimum. Therefore, we presenta hybrid real-coded genetic algorithm and Variable Neighborhood Search(HRCGA-VNS) to solve the problem. VNS is applied after reproductionprocess of GA to repair the offspring and improve GA exploitation capabilitiesin local area to get better result. The test results show that the optimal popsizeof GA is 180, number of generations is 80, combination of cr and mr is 0.7 and0.3 while optimum Kmax of VNS is 40 with number of iterations 50. Eventhough HRCGA-VNS need longer computational time, HRCGA-VNS hasproven to provide a better result based on higher fitness value compared withclassical GA and VNS.

show abstract

Performance-aware scheduling of streaming applications using genetic algorithm

Cited by 24 publications

References 6 publications

Pipelined Dynamic Scheduling of Big Data Streams

Pipelined Dynamic Scheduling of Big Data Streams

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

Hybrid Real-Coded Genetic Algorithm and Variable Neighborhood Search for Optimization of Product Storage

Contact Info

Product

Resources

About