WG-Storm Scheduler for Distributed Stream Processing Engines

Abstract

Stream Processing Engines (SPEs) allow applications to process a large amount of data in real-time. However, to schedule big data applications; the SPEs create several challenges in terms of load balancing, resource utilization, etc. As the volume of data increases over time, it also poses a challenge to predict the resource and application requirements for processing. All these factors play an important role, they can cause problems in achieving maximum throughput due to inefficiency in any of them. Most SPEs ignore the topology, which may minimize throughput during scheduling and may increase network latency. In this paper, we proposed a topology-aware and resource-aware scheduler (named WG-Storm) based on Directed Acyclic Graph (DAG) that enhances the resource usage and overall throughput using efficient tasks assignment. WG-Storm is built on Apache Storm and results are generated using the 2 linear topologies and compared with the 5 state-of-art schedulers including the (A3-Storm, Default, Isolation, Multi-tenant, and Resource-aware). Results have shown up to 30% throughput improvement using minimum resource usage in heterogeneous clusters.