Iterative Scheduling for Distributed Stream Processing Systems

Eskandari, Leila; Mair, Jason; Huang, Zhiyi; Eyers, David

doi:10.1145/3210284.3219768

Cited by 11 publications

(8 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The static strategies work offline and try to assign the tasks to the most suitable nodes in order to minimize the communication latencies between tasks that need to co-operate during the execution of an application. A number of static strategies are topology-aware, such as those listed in [8][9][10], while others are based on resource handling (resource-aware), such as those in [11,12] or [13]. Finally, other recent works employ the idea of linear programming; for example, this is evident in [14][15][16][17].…”

Section: Related Workmentioning

confidence: 99%

Pipelined Dynamic Scheduling of Big Data Streams

Souravlas

Anastasiadou

2020

Applied Sciences

View full text Add to dashboard Cite

We are currently living in the big data era, in which it has become more necessary than ever to develop “smart” schedulers. It is common knowledge that the default Storm scheduler, as well as a large number of static schemes, has presented certain deficiencies. One of the most important of these deficiencies is the weakness in handling cases in which system changes occur. In such a scenario, some type of re-scheduling is necessary to keep the system working in the most efficient way. In this paper, we present a pipeline-based dynamic modular arithmetic-based scheduler (PMOD scheduler), which can be used to re-schedule the streams distributed among a set of nodes and their tasks, when the system parameters (number of tasks, executors or nodes) change. The PMOD scheduler organizes all the required operations in a pipeline scheme, thus reducing the overall processing time.

show abstract

Section: Related Workmentioning

confidence: 99%

Pipelined Dynamic Scheduling of Big Data Streams

Souravlas

Anastasiadou

2020

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Their work assumes that the cluster is homogeneous. Later, they extended their work [28] in heterogeneous clusters and proposed I-Scheduler, an iterative graph partitioning-based heuristic algorithm. This approach finds…”

Section: B Heuristic Approachesmentioning

confidence: 99%

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

et al. 2020

View full text Add to dashboard Cite

Nowadays, there is an accelerating need to efficiently and timely handle large amounts of data that arrives continuously. Streams of big data led to the emergence of several Distributed Stream Processing Systems (DSPS) that assign processing tasks to the available resources (dynamically or not) and route streaming data between them. Efficient scheduling of processing tasks can reduce application latencies and eliminate network congestions. However, the available DSPSs' in-built scheduling techniques are far from optimal. In this work, we extend our previous work [1], where we proposed a linear scheme for the task allocation and scheduling problem. Our scheme takes advantage of pipelines to handle efficiently applications, where there is need for heavy communication (all-to-all) between tasks assigned to pairs of components. In this work, we prove that our scheme is periodic, we provide a communication refinement algorithm and a mechanism to handle many-to-one assignments efficiently. For concreteness, our work is illustrated based on Apache Storm semantics. The performance evaluation depicts that our algorithm achieves load balance and constraints the required buffer space. For throughput testing, we compared our work to the default Storm scheduler, as well as to R-Storm. Our scheme was found to outperform both the other strategies and achieved an average of 25%-40% improvement compared to Storm's default scheduler under different scenarios, mainly as a result of reduced buffering (≈ 45% less memory). Compared to Rstorm, the results indicate an average of 35%-45% improvement.

show abstract

“…As a goal, they try to minimize inter-node communication; by dynamic rescheduling and do not consider the overhead of reassignment of the tasks. I-Scheduler [26] is another scheduling algorithm which tries to overcome the complexity of the task-assignment problem by reducing the total number of tasks. It finds highly communicating tasks by exploiting graph partitioning techniques and take only one instance for each of them.…”

Section: Related Workmentioning

confidence: 99%

A Scheduling Algorithm to Maximize Storm Throughput in Heterogeneous Cluster

Nasiri¹,

Nasehi²,

Arman³

et al. 2020

Preprint

View full text Add to dashboard Cite

In the most popular distributed stream processing frameworks (DSPFs), programs are modeled as a directed acyclic graph. This model allows a DSPF to benefit from the parallelism power of distributed clusters. However, choosing the proper number of vertices for each operator and finding an appropriate mapping between these vertices and processing resources have a determinative effect on overall throughput and resource utilization; while the simplicity of current DSPFs' schedulers leads these frameworks to perform poorly on large-scale clusters. In this paper, we present the design and implementation of a heterogeneity-aware scheduling algorithm that finds the proper number of the vertices of an application graph and maps them to the most suitable cluster node. We start to scale up the application graph over a given cluster gradually, by increasing the topology input rate and taking new instances from bottlenecked vertices. Our experimental results on Storm Micro-Benchmark show that 1) the prediction model estimate CPU utilization with 92% accuracy. 2) Compared to default scheduler of Storm, our scheduler provides 7% to 44% throughput enhancement.3) The proposed method can find the solution within 4% (worst case) of the optimal scheduler which obtains the best scheduling scenario using an exhaustive search on problem design space.

show abstract

Iterative Scheduling for Distributed Stream Processing Systems

Cited by 11 publications

References 7 publications

Pipelined Dynamic Scheduling of Big Data Streams

Pipelined Dynamic Scheduling of Big Data Streams

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

A Scheduling Algorithm to Maximize Storm Throughput in Heterogeneous Cluster

Contact Info

Product

Resources

About