Toward Reliable and Rapid Elasticity for Streaming Dataflows on Clouds

Abstract

The pervasive availability of streaming data is driving interest in distributed Fast Data platforms for streaming applications. Such latency-sensitive applications need to respond to dynamism in the input rates and task behavior using scale-in and -out on elastic Cloud resources. Platforms like Apache Storm do not provide robust capabilities for responding to such dynamism and for rapid task migration across VMs. We propose several dataflow checkpoint and migration approaches that allow a running streaming dataflow to migrate, without any loss of in-flight messages or their internal tasks states, while reducing the time to recover and stabilize. We implement and evaluate these migration strategies on Apache Storm using micro and application dataflows for scaling in and out on up to 2 − 21 Azure VMs. Our results show that we can migrate dataflows of large sizes within 50 sec, in comparison to Storm's default approach that takes over 100 sec. We also find that our approaches stabilize the application much earlier and there is no failure and re-processing of messages.

Dynamic Task Scheduling Scheme for Processing Real-Time Stream Data in Storm Environments

“…In this case, response delay and task loss occur during real-time stream processing of data, which causes failure in task completion within the deadline. For resolving these issues, various scheduling schemes have been examined in which the loads on the worker nodes in a real-time stream environment are considered [10][11][12][13][14][15][16][17][18][19][20][21].…”

“…A study [16] proposed mechanisms to dynamically enact the rescheduling and migration of tasks in a streaming dataflow from one set of virtual machines to another reliably and rapidly. They proposed two task migration strategies such as Drain-Checkpoint-Restore(DCR) and Capture-Checkpoint-Resume(CCR) in Storm by using Redis that is a distributed key/value store.…”

“…Various studies have been conducted for resolving this problem [10][11][12][13][14][15][16][17][18][19][20][21]. In some study, simply calculating the traffic cannot solve the problem of load imbalance on worker nodes.…”

Model-based auto-scaling of distributed data stream processing applications

“…Indeed, by letting multiple instances process the input stream in parallel, operators can efficiently handle load peaks, while avoiding resource wastage in low-load periods. However, operator scaling is particularly challenging, especially in presence of stateful operators, as each parallelism adaptation requires the execution of a reconfiguration protocol to preserve stream and state integrity, often causing significant overhead (see, e.g., [18]). As surveyed in [15], a variety of different techniques have been used to define operator scaling policies, including threshold-based heuristics [6], control theory [4], queueing theory [9], reinforcement learning [7].…”

Minimizing Resource Waste in Heterogeneous Resource Allocation for Data Stream Processing on Clouds

“…The proposed model prioritizes the tasks sent by the federation scheduler with two multi-resource fair scheduling algorithms for cloud and federation. In addition, Shukla et al [34] proposed a mechanism for migrating running streaming dataflow across VMs. Tan et al [35] proposed a Cooperative Coevolution Genetic Programming (CCGP) hyper-heuristic approach.…”