Efficient resource scheduling for the analysis of Big Data streams

Mortazavi-Dehkordi, Mahmood; Zamanifar, Kamran

doi:10.3233/ida-173691

Cited by 8 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mortazavi-Dehkordi & Zamanifar 22 proposed a framework, which examines the topology to determine the output size of its operators for analysis, and partitions. Different scheduling approaches are applied for each.…”

Section: Related Workmentioning

confidence: 99%

WG-Storm Scheduler for Distributed Stream Processing Engines

Ali

Muhammad

Aleem

et al. 2023

Preprint

View full text Add to dashboard Cite

Stream Processing Engines (SPEs) allow applications to process a large amount of data in real-time. However, to schedule big data applications; the SPEs create several challenges in terms of load balancing, resource utilization, etc. As the volume of data increases over time, it also poses a challenge to predict the resource and application requirements for processing. All these factors play an important role, they can cause problems in achieving maximum throughput due to inefficiency in any of them. Most SPEs ignore the topology, which may minimize throughput during scheduling and may increase network latency. In this paper, we proposed a topology-aware and resource-aware scheduler (named WG-Storm) based on Directed Acyclic Graph (DAG) that enhances the resource usage and overall throughput using efficient tasks assignment. WG-Storm is built on Apache Storm and results are generated using the 2 linear topologies and compared with the 5 state-of-art schedulers including the (A3-Storm, Default, Isolation, Multi-tenant, and Resource-aware). Results have shown up to 30% throughput improvement using minimum resource usage in heterogeneous clusters.

show abstract

Section: Related Workmentioning

confidence: 99%

WG-Storm Scheduler for Distributed Stream Processing Engines

Ali

Muhammad

Aleem

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Unlike most approaches that work with Apache Storm, they extended Apache Flink's scheduling workflow. Mortazavi-Dehkordi & Zamanifar [32] combined linear programming with queries' attributes. They proposed Bframework that examines the topology structure of a query to estimate the size of output stream flow of its operators to profile and partition them.…”

Section: Dsp Frameworkmentioning

confidence: 99%

“…Dynamic scheduling techniques ( [5], [9], [24], [32]- [41]) monitor the queue waiting times and performance parameters (e.g. workload, traffic load, system's latency and throughput) during runtime and update tasks' replication degree and their placement (the MAPE-monitor, analyze plan,and executereference model is commonly used for implementing dynamic systems [5], [9] [14]).…”

Section: Dsp Frameworkmentioning

confidence: 99%

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

et al. 2020

View full text Add to dashboard Cite

Nowadays, there is an accelerating need to efficiently and timely handle large amounts of data that arrives continuously. Streams of big data led to the emergence of several Distributed Stream Processing Systems (DSPS) that assign processing tasks to the available resources (dynamically or not) and route streaming data between them. Efficient scheduling of processing tasks can reduce application latencies and eliminate network congestions. However, the available DSPSs' in-built scheduling techniques are far from optimal. In this work, we extend our previous work [1], where we proposed a linear scheme for the task allocation and scheduling problem. Our scheme takes advantage of pipelines to handle efficiently applications, where there is need for heavy communication (all-to-all) between tasks assigned to pairs of components. In this work, we prove that our scheme is periodic, we provide a communication refinement algorithm and a mechanism to handle many-to-one assignments efficiently. For concreteness, our work is illustrated based on Apache Storm semantics. The performance evaluation depicts that our algorithm achieves load balance and constraints the required buffer space. For throughput testing, we compared our work to the default Storm scheduler, as well as to R-Storm. Our scheme was found to outperform both the other strategies and achieved an average of 25%-40% improvement compared to Storm's default scheduler under different scenarios, mainly as a result of reduced buffering (≈ 45% less memory). Compared to Rstorm, the results indicate an average of 35%-45% improvement.

show abstract

“…Both heterogeneous DDSPSs and job applications can have a variety of resource capacities and task complexities, respectively [8]. Consequently, a scheduling approach that does not consider the aspects of resource heterogeneity and task complexity variation in communication and computation might impact the performance and reduce the system frame rate [9][10][11][12].…”

mentioning

confidence: 99%

Job scheduler for streaming applications in heterogeneous distributed processing systems

Al-Sinayyid

Zhu

2020

J Supercomput

View full text Add to dashboard Cite

In this study, we investigated the problem of scheduling streaming applications on a heterogeneous cluster environment and, based on our previous work, developed the maximum throughput scheduler algorithm (MT-Scheduler) for streaming applications. The proposed algorithm uses a dynamic programming technique to efficiently map the application topology onto the heterogeneous distributed system based on computing and data transfer requirements, while also taking into account the capacity of the underlying cluster resources. The proposed approach maximizes the system throughput by identifying and minimizing the time incurred at the computing/transfer bottleneck. The MT-Scheduler supports scheduling applications structured as a directed acyclic graph (DAG). We conducted experiments using three Storm microbenchmark topologies in both simulation and real Apache Storm environments. In terms of the performance evaluation, we compared the proposed MT-Scheduler with the simulated round robin and the default Storm scheduler algorithms. The results indicated that the MT-Scheduler outperforms the default round robin approach in terms of both the average system latency and throughput.

show abstract

Efficient resource scheduling for the analysis of Big Data streams

Cited by 8 publications

References 17 publications

WG-Storm Scheduler for Distributed Stream Processing Engines

WG-Storm Scheduler for Distributed Stream Processing Engines

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

Job scheduler for streaming applications in heterogeneous distributed processing systems

Contact Info

Product

Resources

About