Job scheduler for streaming applications in heterogeneous distributed processing systems

Al-Sinayyid, Ali; Zhu, Mengxia

doi:10.1007/s11227-020-03223-z

Cited by 18 publications

(16 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this case, response delay and task loss occur during real-time stream processing of data, which causes failure in task completion within the deadline. For resolving these issues, various scheduling schemes have been examined in which the loads on the worker nodes in a real-time stream environment are considered [10][11][12][13][14][15][16][17][18][19][20][21].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Task Scheduling Scheme for Processing Real-Time Stream Data in Storm Environments

et al. 2021

View full text Add to dashboard Cite

Owing to the recent advancements in Internet of Things technology, social media, and mobile devices, real-time stream balancing processing systems are commonly used to process vast amounts of data generated in various media. In this paper, we propose a dynamic task scheduling scheme considering task deadlines and node resources. The proposed scheme performs dynamic scheduling using a heterogeneous cluster consisting of various nodes with different performances. Additionally, the loads of the nodes considering the task deadlines are balanced by different task scheduling based on three defined load types. Based on diverse performance evaluations it is shown that the proposed scheme outperforms the conventional schemes.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Various studies have been conducted for resolving this problem [10][11][12][13][14][15][16][17][18][19][20][21]. In some study, simply calculating the traffic cannot solve the problem of load imbalance on worker nodes.…”

Section: Introductionmentioning

confidence: 99%

Dynamic Task Scheduling Scheme for Processing Real-Time Stream Data in Storm Environments

et al. 2021

View full text Add to dashboard Cite

show abstract

“…A number of static strategies are topology-aware, such as those listed in [8][9][10], while others are based on resource handling (resource-aware), such as those in [11,12] or [13]. Finally, other recent works employ the idea of linear programming; for example, this is evident in [14][15][16][17].…”

Section: Related Workmentioning

confidence: 99%

Pipelined Dynamic Scheduling of Big Data Streams

Souravlas

Anastasiadou

2020

Applied Sciences

View full text Add to dashboard Cite

We are currently living in the big data era, in which it has become more necessary than ever to develop “smart” schedulers. It is common knowledge that the default Storm scheduler, as well as a large number of static schemes, has presented certain deficiencies. One of the most important of these deficiencies is the weakness in handling cases in which system changes occur. In such a scenario, some type of re-scheduling is necessary to keep the system working in the most efficient way. In this paper, we present a pipeline-based dynamic modular arithmetic-based scheduler (PMOD scheduler), which can be used to re-schedule the streams distributed among a set of nodes and their tasks, when the system parameters (number of tasks, executors or nodes) change. The PMOD scheduler organizes all the required operations in a pipeline scheme, thus reducing the overall processing time.

show abstract

“…Towards this direction, Al-Sinayyid and Zhu [30] proposed MT-scheduler that uses a dynamic programming technique to efficiently map a DAG onto heterogeneous systems. They proposed a polynomial-time heuristic solution that is based on computing and data transfer requirements, and the capacity of the underlying cluster resources.…”

Section: Dsp Frameworkmentioning

confidence: 99%

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

et al. 2020

View full text Add to dashboard Cite

Nowadays, there is an accelerating need to efficiently and timely handle large amounts of data that arrives continuously. Streams of big data led to the emergence of several Distributed Stream Processing Systems (DSPS) that assign processing tasks to the available resources (dynamically or not) and route streaming data between them. Efficient scheduling of processing tasks can reduce application latencies and eliminate network congestions. However, the available DSPSs' in-built scheduling techniques are far from optimal. In this work, we extend our previous work [1], where we proposed a linear scheme for the task allocation and scheduling problem. Our scheme takes advantage of pipelines to handle efficiently applications, where there is need for heavy communication (all-to-all) between tasks assigned to pairs of components. In this work, we prove that our scheme is periodic, we provide a communication refinement algorithm and a mechanism to handle many-to-one assignments efficiently. For concreteness, our work is illustrated based on Apache Storm semantics. The performance evaluation depicts that our algorithm achieves load balance and constraints the required buffer space. For throughput testing, we compared our work to the default Storm scheduler, as well as to R-Storm. Our scheme was found to outperform both the other strategies and achieved an average of 25%-40% improvement compared to Storm's default scheduler under different scenarios, mainly as a result of reduced buffering (≈ 45% less memory). Compared to Rstorm, the results indicate an average of 35%-45% improvement.

show abstract

Job scheduler for streaming applications in heterogeneous distributed processing systems

Cited by 18 publications

References 41 publications

Dynamic Task Scheduling Scheme for Processing Real-Time Stream Data in Storm Environments

Dynamic Task Scheduling Scheme for Processing Real-Time Stream Data in Storm Environments

Pipelined Dynamic Scheduling of Big Data Streams

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

Contact Info

Product

Resources

About