Load Distribution for Distributed Stream Processing

Xing, Ying

doi:10.1007/978-3-540-30192-9_11

Cited by 6 publications

(10 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another recent paper that comes out from Aurora project which is very relevant to this paper is the dynamic load distribution strategy [22]. By and large, the goal of query optimization is to map operators efficiently to resources in a distributed environment.…”

Section: Related Workmentioning

confidence: 99%

“…As a result, some streams may get processed significantly slower than the rest. Most previous work on streaming data processing assign operators to each processing unit in a static or semi-dynamic fashion [4,6,13,21,22]. By doing worst case analysis at compilation and/or deployment, the system may be over-provisioned but has a reasonable chance to make the desired service level.…”

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

Active workflow system for near real-time extreme-scale science

Zhang

Liu

Klasky

et al. 2014

Proceedings of the First Workshop on Parallel Programming for Analytics Applications

View full text Add to dashboard Cite

In recent years, streaming-based data processing has been gaining substantial traction for dealing with overwhelming data generated by real-time applications, from both enterprise sources and scientific computing. In this work, however, we look at an emerging class of scientific data with Near Real-Time (NRT) requirement, in which data is typically generated in a bursty fashion with the near real-time constraints being applied primarily between bursts, rather than within a stream. A key challenge for this types of data sources is that the processing time per data element is not uniform, and not always feasible to predict. Given the observations on the increasing unpredictability of compute load and system dynamics, this work looks to adapt streamingbased approach to the context of this new class of large experiments and simulations that have complex run-time control and analysis issues.In particular, we deploy a novel two-tier scheme for handling the increasing unpredictability of runtime behaviors: Instead of relying on determining what and where to run the scientific workflows beforehand or partial dynamically, the decision will also be adaptively enhanced online according to system runtime status. This is enabled by embedding workflow along with data streams. Specifically, we break data outputs generated from experiments or simulations into multiple self-describing "chunks", which we call active data objects. As such, if there is a transient hotspot observed, a Permission to make digital data object with unfinished workflow pipeline can break its previous schedule and search for a least loaded location to continue the execution. Our preliminary experiment results based on synthetic workloads demonstrate the proposed active workflow system as a very promising solution by outperforming the state-of-the-art semi-dynamic workflow schedulers with an improved workflow completion time, as well as a good scalability.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 98%

Active workflow system for near real-time extreme-scale science

Zhang

Liu

Klasky

et al. 2014

Proceedings of the First Workshop on Parallel Programming for Analytics Applications

View full text Add to dashboard Cite

show abstract

“…In [36] a decentralized scheme based on load coalescing is proposed. This work has a similar aim with our work for minimizing communication delays, however, it takes a different direction as we focus on a centralized approach with a fast allocation algorithm.…”

Section: Related Workmentioning

confidence: 99%

“…We aim to reduce the communication latency while keeping the nodes below a computational load threshold. Although previous work [11,36,26] suggests decentralized task allocation schemes for avoiding bottlenecks, we argue that a centralized approach can be efficient, as long as we design fast allocation algorithms. We thus devise on fast task allocation heuristics, while keeping the quality of the resulting allocation high.…”

Section: Introductionmentioning

confidence: 99%

Fast Heuristics for Near-Optimal Task Allocation in Data Stream Processing over Clusters

Chatzistergiou

Viglas

2014

Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

View full text Add to dashboard Cite

We study provisioning and job reconfiguration techniques for adapting to execution environment changes when processing data streams on cluster-based deployments. By monitoring the performance of an executing job, we identify computation and communication bottlenecks. In such cases we reconfigure the job by reallocating its tasks to minimize the communication cost. Our work targets dataintensive applications where the inter-node transfer latency is significant. We aim to minimize the transfer latency while keeping the nodes below some computational load threshold. We propose a scalable centralized scheme that employs fast allocation heuristics. Our techniques are based on a general group-based job representation that is commonly found in many distributed data stream processing frameworks. Using this representation we devise lineartime task allocation algorithms that improve existing quadratictime solutions in practical cases. We have implemented and evaluated our proposals using both synthetic and real-world scenarios. Our results show that our algorithms: (a) exhibit significant allocation throughput while producing near-optimal allocations, and (b) significantly improve existing task-level approaches.

show abstract

“…As in [11], [12] assumes homogeneous machines and perfect networks; on the other hand, it uses CPU capacity to reflect resource constraint and operator clustering as a preprocessing step to prevent costly data crossing the network. [10] attempts to consider the situations where the network transfer delay cannot be ignored by grouping neighboring operators in the initial mapping. When adjustment is necessary, only operators at the boundary are migrated to the neighboring host, so as to avoid creating excessive network traffic.…”

Section: Related Workmentioning

confidence: 99%

Potential-driven load distribution for distributed data stream processing

Wang

Sharaf

Guo

et al. 2008

Proceedings of the 2nd International Workshop on Scalable Stream Processing System

View full text Add to dashboard Cite

A large class of applications require real-time processing of continuous stream data resulting in the development of data stream management systems (DSMS). Since many of these applications are distributed, distributed DSMSs are starting to receive attention. In this paper, we focus on an important issue in distributed DSMS operation, namely load distribution to minimize end-to-end latency. We identify the often conflicting requirements of load distribution, and propose a "potential-driven" load distribution approach to mimic the movements of objects in the physical world. Our approach also takes into account heterogeneous machines, different network conditions, and resource constraints. We present experimental results that investigate our algorithms from various aspects, and show that they outperform existing techniques in terms of end-to-end latency.

show abstract

Load Distribution for Distributed Stream Processing

Cited by 6 publications

References 9 publications

Active workflow system for near real-time extreme-scale science

Active workflow system for near real-time extreme-scale science

Fast Heuristics for Near-Optimal Task Allocation in Data Stream Processing over Clusters

Potential-driven load distribution for distributed data stream processing

Contact Info

Product

Resources

About