Model-driven scheduling for distributed stream processing systems

Shukla, Aparna; Simmhan, Yogesh

doi:10.1016/j.jpdc.2018.02.003

Cited by 41 publications

(28 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Without loss of generality, we assume that the tuple processing time of all the tasks is almost the same (this is a logical assumption used also by shuffle grouping, see [8], [14]). Under this hypothesis, we can divide the overall processing into a set of well-defined processing steps and stages, terms that are defined in the following subsection.…”

Section: A Preliminaries In Stormmentioning

confidence: 99%

“…A dynamic scheme could adopt data parallelism and scale out the number of parallel instances for the operator that is overloaded and becomes a bottleneck and/or increase the number of VMs that run in the cluster [14]. Possible task migrations would be needed to reduce resource utilization imbalances between nodes.…”

Section: Bitmapmentioning

confidence: 99%

“…They are designed to execute complex streaming applications as Directed Acyclic Graphs (DAGs) over tuples of a stream. They leverage data parallelism using mutiple threads of execution per task (replicas), in addition to pipelined and task-parallel execution of the desired DAG (see subsection II.A "Preliminaries in Storm") [14].…”

mentioning

confidence: 99%

See 2 more Smart Citations

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

et al. 2020

View full text Add to dashboard Cite

Nowadays, there is an accelerating need to efficiently and timely handle large amounts of data that arrives continuously. Streams of big data led to the emergence of several Distributed Stream Processing Systems (DSPS) that assign processing tasks to the available resources (dynamically or not) and route streaming data between them. Efficient scheduling of processing tasks can reduce application latencies and eliminate network congestions. However, the available DSPSs' in-built scheduling techniques are far from optimal. In this work, we extend our previous work [1], where we proposed a linear scheme for the task allocation and scheduling problem. Our scheme takes advantage of pipelines to handle efficiently applications, where there is need for heavy communication (all-to-all) between tasks assigned to pairs of components. In this work, we prove that our scheme is periodic, we provide a communication refinement algorithm and a mechanism to handle many-to-one assignments efficiently. For concreteness, our work is illustrated based on Apache Storm semantics. The performance evaluation depicts that our algorithm achieves load balance and constraints the required buffer space. For throughput testing, we compared our work to the default Storm scheduler, as well as to R-Storm. Our scheme was found to outperform both the other strategies and achieved an average of 25%-40% improvement compared to Storm's default scheduler under different scenarios, mainly as a result of reduced buffering (≈ 45% less memory). Compared to Rstorm, the results indicate an average of 35%-45% improvement.

show abstract

Section: A Preliminaries In Stormmentioning

confidence: 99%

Section: Bitmapmentioning

confidence: 99%

See 1 more Smart Citation

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

et al. 2020

View full text Add to dashboard Cite

show abstract

“…In particular, the latter drawback is strongest for multimedia data, for example, in applications that use video and image acquisition devices. For that reason, it is difficult to implement a centralized multimedia analysis system in the cloud [ 9 , 10 ].…”

Section: Introductionmentioning

confidence: 99%

Collaborative Working Architecture for IoT-Based Applications

Mora

Pont

Gil

et al. 2018

Sensors

View full text Add to dashboard Cite

The new sensing applications need enhanced computing capabilities to handle the requirements of complex and huge data processing. The Internet of Things (IoT) concept brings processing and communication features to devices. In addition, the Cloud Computing paradigm provides resources and infrastructures for performing the computations and outsourcing the work from the IoT devices. This scenario opens new opportunities for designing advanced IoT-based applications, however, there is still much research to be done to properly gear all the systems for working together. This work proposes a collaborative model and an architecture to take advantage of the available computing resources. The resulting architecture involves a novel network design with different levels which combines sensing and processing capabilities based on the Mobile Cloud Computing (MCC) paradigm. An experiment is included to demonstrate that this approach can be used in diverse real applications. The results show the flexibility of the architecture to perform complex computational tasks of advanced applications.

show abstract

“…A related but separate problem is to determine the new resource allocation for the dataflow (number and sizes of VMs) and the new mapping of its tasks onto the VMs. This is outside the scope of this paper, but has been examined elsewhere [4]. Having a new schedule is a precursor to the dynamic enactment of the schedule, which we target in the current paper.…”

Section: Introductionmentioning

confidence: 99%

Toward Reliable and Rapid Elasticity for Streaming Dataflows on Clouds

Shukla¹,

Simmhan

2018

2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)

Self Cite

View full text Add to dashboard Cite

The pervasive availability of streaming data is driving interest in distributed Fast Data platforms for streaming applications. Such latency-sensitive applications need to respond to dynamism in the input rates and task behavior using scale-in and -out on elastic Cloud resources. Platforms like Apache Storm do not provide robust capabilities for responding to such dynamism and for rapid task migration across VMs. We propose several dataflow checkpoint and migration approaches that allow a running streaming dataflow to migrate, without any loss of in-flight messages or their internal tasks states, while reducing the time to recover and stabilize. We implement and evaluate these migration strategies on Apache Storm using micro and application dataflows for scaling in and out on up to 2 − 21 Azure VMs. Our results show that we can migrate dataflows of large sizes within 50 sec, in comparison to Storm's default approach that takes over 100 sec. We also find that our approaches stabilize the application much earlier and there is no failure and re-processing of messages.

show abstract

Model-driven scheduling for distributed stream processing systems

Cited by 41 publications

References 47 publications

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

Pipeline-Based Linear Scheduling of Big Data Streams in the Cloud

Collaborative Working Architecture for IoT-Based Applications

Toward Reliable and Rapid Elasticity for Streaming Dataflows on Clouds

Contact Info

Product

Resources

About