Workload scheduling in distributed stream processors using graph partitioning

Fischer, Lorenz; Bernstein, Abraham

doi:10.1109/bigdata.2015.7363749

Cited by 29 publications

(25 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On-demand computing resource allocation is the main target accomplished by adjusting the task schedule of the edge gateway via a lightweight virtualization technology (i.e., Docker). The authors of [21] trying to respond to question of how to distribute workload to available machines propose a workload scheduling strategy that is based on a graph partitioning algorithm. The proposed scheduler is application agnostic and builds on the data related to the communication behavior of running applications.…”

Section: Related Workmentioning

confidence: 99%

A probabilistic model for assigning queries at the edge

Kolomvatsos

Anagnostopoulos

2019

Computing

View full text Add to dashboard Cite

Data management at the edge of the network can increase the performance of applications as the processing is realized close to end users limiting the observed latency in the provision of responses. A typical data processing involves the execution of queries/tasks defined by users or applications asking for responses in the form of analytics. Query/task execution can be realized at the edge nodes that can undertake the responsibility of delivering the desired analytics to the interested users or applications. In this paper, we deal with the problem of allocating queries to a number of edge nodes. The aim is to support the goal of eliminating further the latency by allocating queries to nodes that exhibit a low load and high processing speed, thus, they can respond in the minimum time. Before any allocation, we propose a method for estimating the computational burden that a query/task will add to a node and, afterwards, we proceed with the final assignment. The allocation is concluded by the assistance of an ensemble similarity scheme responsible to deliver the complexity class for each query/task and a probabilistic decision making model. The proposed scheme matches the characteristics of the incoming queries and edge nodes trying to conclude the optimal allocation. We discuss our mechanism and through a large set of simulations and the adoption of benchmarking queries, we reveal the potentials of the proposed model supported by numerical results.

show abstract

Section: Related Workmentioning

confidence: 99%

A probabilistic model for assigning queries at the edge

Kolomvatsos

Anagnostopoulos

2019

Computing

View full text Add to dashboard Cite

show abstract

“…Acking is done for all events, and the checkpoint interval is periodic (30 secs, by default) and has to be configured to balance operational costs and rollback loss for a dataflow. Hence, they also pose additional overheads if the fault-tolerance is a concern only during active migration and not during regular operations [8], [9]. This can be punitive during normal operations if the input rates are high [10].…”

Section: Background and Motivationmentioning

confidence: 99%

Toward Reliable and Rapid Elasticity for Streaming Dataflows on Clouds

Shukla¹,

Simmhan

2018

2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

The pervasive availability of streaming data is driving interest in distributed Fast Data platforms for streaming applications. Such latency-sensitive applications need to respond to dynamism in the input rates and task behavior using scale-in and -out on elastic Cloud resources. Platforms like Apache Storm do not provide robust capabilities for responding to such dynamism and for rapid task migration across VMs. We propose several dataflow checkpoint and migration approaches that allow a running streaming dataflow to migrate, without any loss of in-flight messages or their internal tasks states, while reducing the time to recover and stabilize. We implement and evaluate these migration strategies on Apache Storm using micro and application dataflows for scaling in and out on up to 2 − 21 Azure VMs. Our results show that we can migrate dataflows of large sizes within 50 sec, in comparison to Storm's default approach that takes over 100 sec. We also find that our approaches stabilize the application much earlier and there is no failure and re-processing of messages.

show abstract

“…Fisher et al [11] solve the scheduling problem using graph partitioning. POIs are vertices of the graph, and are weighted by the computational resources they consume.…”

Section: Operator Instance Schedulingmentioning

confidence: 99%

“…Any online scheduler that actively measures communication between POIs can then notice the improvement and re-visit the POI placement decision, leading to even better performance. Our approach is similar to [11] as it relies on Metis for graph partitioning. Instead of considering a graph of POIs communicating, we consider a graph of keys that cooccur in the data.…”

Section: Operator Instance Schedulingmentioning

confidence: 99%

Locality-Aware Routing in Stateful Streaming Applications

Caneill

Rheddane

Leroy

et al. 2016

Proceedings of the 17th International Middleware Conference

View full text Add to dashboard Cite

International audienceDistributed stream processing engines continuously execute series of operators on data streams. Horizontal scaling is achieved by deploying multiple instances of each operator in order to process data tuples in parallel. As the application is distributed on an increasingly high number of servers, the likelihood that the stream is sent to a different server for each operator increases. This is particularly important in the case of stateful applications that rely on keys to deterministically route messages to a specific instance of an operator. Since network is a bottleneck for many stream applications, this behavior significantly degrades their performance. Our objective is to improve stream locality for stateful stream processing applications. We propose to analyse traces of the application to uncover correlations between the keys used in successive routing operations. By assigning correlated keys to instances hosted on the same server, we significantly reduce network consumption and increase performance while preserving load balance. Furthermore, this approach is executed online, so that the assignment can automatically adapt to changes in the characteristics of the data. Data migration is handled seamlessly with each routing configuration update. We implemented and evaluated our protocol using Apache Storm, with a real workload consisting of geo-tagged Flickr pictures as well as Twitter publications. Our results show a significant improvement in throughput

show abstract

Workload scheduling in distributed stream processors using graph partitioning

Cited by 29 publications

References 20 publications

A probabilistic model for assigning queries at the edge

A probabilistic model for assigning queries at the edge

Toward Reliable and Rapid Elasticity for Streaming Dataflows on Clouds

Locality-Aware Routing in Stateful Streaming Applications

Contact Info

Product

Resources

About