AdaStorm: Resource Efficient Storm with Adaptive Configuration

Weng, Zujian; Guo, Qi; Wang, Chunkai; Meng, Xiaofeng; He, Bingsheng

doi:10.1109/icde.2017.178

Cited by 11 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… Weng et al (2017) presented AdaStorm to solve the static topology configuration issue of the default scheduler. With the help of a machine learning algorithm, AdaStorm dynamically adjusts the topology configuration according to the communication traffic.…”

Section: Literature Reviewmentioning

confidence: 99%

“…This over-provisioning may cause low resource utilization and higher communication cost. In addition, the application’s computational requirement is overlooked which reduces throughput ( Madsen & Zhou, 2015 ; Peng et al, 2015 ; Eskandari, Huang & Eyers, 2016 ; Weng et al, 2017 ). The default Storm is designed for the homogeneous cluster which may introduce performance bottleneck problem ( Liu, 2017 ; Smirnov, Melnik & Nasonov, 2017 ; Zhang et al, 2017 ).…”

Section: Literature Reviewmentioning

confidence: 99%

“…Table 2 shows the results generated by different schedulers proposed in this domain. Weng et al (2017) considered CPU and Memory while scheduling. Only 15% improvement in CPU use is achieved; however, MF-Storm saved up to 66% computational resources of the cluster.…”

Section: Experimental Evaluationmentioning

confidence: 99%

“…Similarly, some techniques ( Fischer & Bernstein, 2015 ; Smirnov, Melnik & Nasonov, 2017 ; Weng et al, 2017 ; Gulzar Ahmad et al, 2020 ; Elahi et al, 2022 ) are proposed for homogenous clusters only and are unable to assign workload according to the node's computing power (heterogeneity). In recent years, resource-aware schedulers ( Peng et al, 2015 ; Xue et al, 2015 ; Weng et al, 2017 ; Liu & Buyya, 2018 ; Eskandari et al, 2018b ; Al-Sinayyid & Zhu, 2020 ; Muhammad & Aleem, 2021 ; Muhammad, Aleem & Islam, 2021 ; Farooq et al, 2022 ) are also proposed which are either non-adaptive or static in nature and cannot accommodate real-time (dynamic) changes. Moreover, CPU, Memory, and bandwidth are considered a resource by existing resource-aware schedulers, however, they do not truly represent the computation power of a node (“FLOPS (Floating Point Operations Per Second) Definition”, ).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase throughput

Muhammad¹,

Qadir²

2022

PeerJ Computer Science

View full text Add to dashboard Cite

Background A scheduling algorithm tries to schedule multiple computational tasks on a cluster of multiple computing nodes to maximize throughput with optimal utilization of computational and communicational resources. A Stream Processing Engine (SPE) is deployed to run streaming applications (computational tasks) on a computational cluster which helps execution and coordination of these applications. It is observed that there is a gap in the optimal mapping of a computational and communicational load of a streaming application on the underlying computational and communication power of the resources (cluster). Frequently communicated tasks are scheduled at different processing nodes with relatively slow communicating links. This increases network latency with a decrease in resource utilization. Hence, reduces the achieved throughput of the cluster significantly. Methods MF-Storm, a max-flow min-cut based job scheduler is presented to achieve a near-optimum schedule to maximize throughput. It schedules a streaming application by considering the processing, communication demands, available computational and communicational resources in a heterogeneous cluster, dynamically with minimized scheduling cost. To keep the scheduling cost minimum, the scheduler is built in a pipeline with two major stages: in the first stage, the application’s tasks graph is partitioned using the max-flow min-cut algorithm to minimize inter-partition traffic, and in the second stage, these partitions are assigned to computing nodes according to the computational power of the cluster’s nodes. Results Extensive experiments were done to evaluate the performance of MF-Storm using different topologies with multiple scenarios on a physical cluster implementation. Results showed on average 148% improvement in throughput with 30% less computational resources as compared to different state-of-the-art schedulers.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Section: Experimental Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase throughput

Muhammad¹,

Qadir²

2022

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…T-Storm [13] requires that each node has only one available worker for each topology to avoid the inter-worker traffic inside one node which is only suitable for lightly loaded topologies. There is a fair number of research work on addressing heterogeneity in terms of resource and input rate in DSP [9,12,33,34]. For instance, Buddhika et al [33] introduce an on-line scheduler to mitigate the impact of interference on the performance of DSP.…”

Section: B Operator Placement and Task Scheduling In Dspmentioning

confidence: 99%

TurboStream: Towards Low-Latency Data Stream Processing

Liu

Ibrahim

et al. 2018

2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

Data Stream Processing (DSP) applications are often modelled as a directed acyclic graph: operators with data streams among them. Inter-operator communications can have a significant impact on the latency of DSP applications, accounting for 86% of the total latency. Despite their impact, there has been relatively little work on optimizing interoperator communications, focusing on reducing inter-node traffic but not considering inter-process communication (IPC) inside a node, which often generates high latency due to the multiple memory-copy operations. This paper describes the design and implementation of TurboStream, a new DSP system designed specifically to address the high latency caused by inter-operator communications. To achieve this goal, we introduce (1) an improved IPC framework with OSRBuffer, a DSP-oriented buffer, to reduce memory-copy operations and waiting time of each single message when transmitting messages between the operators inside one node, and (2) a coarse-grained scheduler that consolidates operator instances and assigns them to nodes to diminish the inter-node IPC traffic. Using a prototype implementation, we show that our improved IPC framework reduces the end-to-end latency of intra-node IPC by 45.64% to 99.30%. Moreover, TurboStream reduces the latency of DSP by 83.23% compared to JStorm.

show abstract

Dynamic energy-efficient scheduling for streaming applications in storm

Dai

Liu

et al. 2021

Computing

View full text Add to dashboard Cite

AdaStorm: Resource Efficient Storm with Adaptive Configuration

Cited by 11 publications

References 3 publications

MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase throughput

MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase throughput

TurboStream: Towards Low-Latency Data Stream Processing

Dynamic energy-efficient scheduling for streaming applications in storm

Contact Info

Product

Resources

About