State access patterns in stream parallel computations

Danelutto, Marco; Kilpatrick, Peter; Mencagli, Gabriele; Torquati, Massimo

doi:10.1177/1094342017694134

Cited by 5 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A practical example of applying such a scheme is provided by Hochreiner et al [69] in their distributed SP system PESP. Finally, Danelu o et al [32] provide a systematic classi cation of state access pa erns in SP systems.…”

Section: Infrastructurementioning

confidence: 99%

See 1 more Smart Citation

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

Stream Processing (SP) has evolved as the leading paradigm to process and gain value from the high volume of streaming data produced e.g. in the domain of the Internet of ings. An SP system is a middleware that deploys a network of operators between data sources, such as sensors, and the consuming applications. SP systems typically face intense and highly dynamic data streams. Parallelization and elasticity enables SP systems to process these streams with continuously high quality of service. e current research landscape provides a broad spectrum of methods for parallelization and elasticity in SP. Each method makes speci c assumptions and focuses on particular aspects of the problem. However, the literature lacks a comprehensive overview and categorization of the state of the art in SP parallelization and elasticity, which is necessary to consolidate the state of the research and to plan future research directions on this basis. erefore, in this survey, we study the literature and develop a classi cation of current methods for both parallelization and elasticity in SP systems. or even di erent processing nodes in a shared-nothing cloud-based infrastructure. Frequent state synchronization must not hamper parallel processing, while the processing results have to remain consistent. Research proposes di erent approaches for parallel, stateless and stateful SP. ey di er in assumptions about the operator functions and state externalization mechanisms an SP system supports.is led to the development of a broad range of parallelization approaches tackling di erent problem cases.Second, how to continuously adapt the level of parallelization when the conditions of the SP operators, e.g. the workload or resources available, change at runtime. On the one hand, an SP system always needs enough resources to process the input data streams with a satisfying quality of service (QoS), e.g. latency or throughput. On the other hand, continuous provisioning of computing resources for peak workloads wastes resources at o -peak hours. us, an elastic SP system scales its resources according to the current need. Cloud computing provides on-demand resources to realize such elasticity [9]. e pay-as-you-go business model of cloud computing allows to cut costs by dynamically adapting the resource reservations to the needs of the SP system. It is challenging to strive the right balance between resource over-provisioning-which is costly, but is robust to workload uctuations-and on-demand scaling-which is cheap, but is vulnerable to sudden workload peaks. To this end, academia and industry developed elasticity methods. Again, they di er in their optimization objectives and assumptions about the operator parallelization model employed, the target system architecture, state management as well as timing and methodology.While there are many works that propose methods and solutions for speci c parallelization and elasticity problems in SP systems, there is a severe lack of overview, comparison, and classi cation of these methods. When we investigate...

show abstract

Section: Infrastructurementioning

confidence: 99%

“…A stateful operator can implement shu e grouping if all instances can access the complete operator state. e order of state access shall unrestricted to avoid sequential processing [32].…”

Section: Datamentioning

confidence: 99%

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…In [28,37] a description of the suitable techniques is provided by focusing on locking algorithms and scheduling strategies that preserve the computation correctness for different kinds of stateful computations. Although interesting such papers do not focus on the nature of the state, but are based on generic descriptions of what the computation semantics expects from the output results.…”

Section: Related Workmentioning

confidence: 99%

Harnessing sliding-window execution semantics for parallel stream processing

Mencagli

Torquati

Fabio

et al. 2018

Journal of Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

According to the recent trend in data acquisition and processing technology, big data are increasingly available in the form of unbounded streams of elementary data items to be processed in real-time. In this paper we study in detail the paradigm of sliding windows, a well-known technique for approximated queries that update their results continuously as new fresh data arrive from the stream. In this work we focus on the relationship between the various existing sliding window semantics and the way the query processing is performed from the parallelism perspective. From this study two alternative parallel models are identified, each covering semantics with very precise properties. Each model is described in terms of its pros and cons, and parallel implementations in the FastFlow framework are analyzed by discussing the layout of the concurrent data structures used for the efficient windows representation in each model.

show abstract

“…Examples of these patterns are pipelines, farms, and data stream-based skeletons. Danelutto et al (2017) introduce a set of state access patterns suitable for managing accesses to states in parallel computations operating on streams. The state access patterns are useful for modelling typical stream parallel applications.…”

Section: Special Issue Contentsmentioning

confidence: 99%

High-level programming for heterogeneous and hierarchical parallel systems

Blas

Brown

2018

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

High-Level Heterogeneous and Hierarchical Parallel Systems (HLPGPU) aims to bring together researchers and practitioners to present new results and ongoing work on those aspects of high-level programming relevant, or specific to GPGPUs and new architectures. The 2016 HLPGPU symposium was an event co-located with the HiPEAC conference in Prague, Czech Republic. HLPGPU is targeted at high-level parallel techniques, including programming models, libraries and languages, algorithmic skeletons, refactoring tools and techniques for parallel patterns, tools and systems to aid parallel programming, heterogeneous computing, timing analysis and statistical performance models.

show abstract

State access patterns in stream parallel computations

Cited by 5 publications

References 20 publications

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Harnessing sliding-window execution semantics for parallel stream processing

High-level programming for heterogeneous and hierarchical parallel systems

Contact Info

Product

Resources

About