Partitioning functions for stateful data parallelism in stream processing

Gedik, Buğra

doi:10.1007/s00778-013-0335-9

Cited by 102 publications

(81 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The state migration must be performed by respecting the computation semantics (state partitions must be accessed atomically) and must not violate the strict sequential order in which items of the same group must be processed. Techniques to solve this issue in the case of autonomic/elastic solutions have been proposed in some past research works [9,15]. Symmetric actions are taken in the case of parallelism degree reduction.…”

Section: Fpsapmentioning

confidence: 99%

State access patterns in stream parallel computations

Danelutto

Kilpatrick²,

Mencagli

et al. 2017

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

We introduce a set of state access patterns suitable for managing accesses to state in parallel computations operating on streams. The state access patterns are useful for modelling typical stream parallel applications. We present a classification of the patterns according to the extent and way in which the state can be structured and accessed. We define precisely the state access patterns and discuss possible implementation schemas, performances and possibilities to manage adaptivity (parallelism degree) in the patterns. We present experimental results relative to implementations built on top of the structured parallel programming framework FastFlow that demonstrate the feasibility and efficiency of the proposed access patterns.

show abstract

Section: Fpsapmentioning

confidence: 99%

State access patterns in stream parallel computations

Danelutto

Kilpatrick²,

Mencagli

et al. 2017

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…To overcome this problem, a plethora of Adaptive Query Processing (AQP) techniques have been recently proposed in the literature aiming to adapt the runtime query plan in respond to changes in the execution environment or the characteristics of the streaming data [4][5][6][7][8]. The rationale followed by these AQP techniques can be condensed into a three-phase procedure, called adaptivity loop [9].…”

Section: Main Textmentioning

confidence: 99%

“…where G 1 and G 2 are given by (4) and Beta is the beta function [21]. Theα andβ estimates can then be found by minimizing Eq.…”

Section: Appendixmentioning

confidence: 99%

See 1 more Smart Citation

Incorporating change detection in the monitoring phase of adaptive query processing

Tsamoura

Gounaris

Manolopoulos

2016

J Internet Serv Appl

View full text Add to dashboard Cite

Recent Big Data research typically emphasizes on the need to address the challenges stemming from the volume, velocity, variety and veracity aspects. However, another cross-cutting property of Big Data is volatility. In database technology, volatility is addressed with the help of adaptive query processing (AQP), which has become the dominant paradigm for executing queries in dynamic and/or streaming environments. As the characteristics of the runtime environment may vary significantly along time, AQP techniques employ a three-phase adaptivity loop to process the input queries, comprising feedback collection, analysis and re-optimization. In the monitoring phase, the standard approach is to collect feedback in a fixed-size sliding window. However, several problems arise when the techniques adopt a fixed-size sliding window for maintaining runtime collected feedback. In this work, we tackle this limitation and we propose a novel monitoring phase, which assesses the collected feedback rendering an AQP technique capable of taking more informed decisions during the subsequent phases. The proposed approach is non-intrusive to the state-of-the-art adaptivity loop and can adopt any state-of-the-art online change detection algorithm through its plug-and-play abstraction. Another contribution of this work is a novel algorithm for detecting changes in a filter's drop probability, called β-CUSUM. The potential of the novel monitoring phase and of β-CUSUM is experimentally evaluated using both real-world and synthetic datasets.

show abstract

“…A recent paper [11] discusses how to define a partitioning function, which can balance load, memory and bandwidth, while ensuring changes to the partitioning function impacts as small a subset of keys as possible, in order to minimize the need for state migration. The method of "The Power of Two Choices" (PoTC) [29] continuously defines two hash functions h1(x) and h2(x), such that each key x can be sent to one of two alternative downstream operator instances.…”

Section: Adaptive Schedulermentioning

confidence: 99%

Integrative Dynamic Reconfiguration in a Parallel Stream Processing Engine

Madsen

Zhou

Cao

2017

2017 IEEE 33rd International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

Load balancing, operator instance collocations and horizontal scaling are critical issues in Parallel Stream Processing Engines to achieve low data processing latency, optimized cluster utilization and minimized communication cost respectively. In previous work, these issues are typically tackled separately and independently. We argue that these problems are tightly coupled in the sense that they all need to determine the allocations of workloads and migrate computational states at runtime. Optimizing them independently would result in suboptimal solutions. Therefore, in this paper, we investigate how these three issues can be modeled as one integrated optimization problem. In particular, we first consider jobs where workload allocations have little effect on the communication cost, and model the problem of load balance as a Mixed-Integer Linear Program. Afterwards, we present an extended solution called ALBIC, which support general jobs. We implement the proposed techniques on top of Apache Storm, an open-source Parallel Stream Processing Engine. The extensive experimental results over both synthetic and real datasets show that our techniques clearly outperform existing approaches.

show abstract

Partitioning functions for stateful data parallelism in stream processing

Cited by 102 publications

References 21 publications

State access patterns in stream parallel computations

State access patterns in stream parallel computations

Incorporating change detection in the monitoring phase of adaptive query processing

Integrative Dynamic Reconfiguration in a Parallel Stream Processing Engine

Contact Info

Product

Resources

About