Self-adaptive processing graph with operator fission for elastic stream processing

Hidalgo, Nicolás; Wladdimiro, Daniel; Rosas, Erika

doi:10.1016/j.jss.2016.06.010

Cited by 44 publications

(23 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hidalgo et al [63] propose a hybrid reactive and proactive elasticity controller implemented on top of the S4 SP system [110]. e elasticity controller has two parts, a reactive short-term adaptation and a proactive mid-term adaptation.…”

Section: Centralized Elasticitymentioning

confidence: 99%

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

Stream Processing (SP) has evolved as the leading paradigm to process and gain value from the high volume of streaming data produced e.g. in the domain of the Internet of ings. An SP system is a middleware that deploys a network of operators between data sources, such as sensors, and the consuming applications. SP systems typically face intense and highly dynamic data streams. Parallelization and elasticity enables SP systems to process these streams with continuously high quality of service. e current research landscape provides a broad spectrum of methods for parallelization and elasticity in SP. Each method makes speci c assumptions and focuses on particular aspects of the problem. However, the literature lacks a comprehensive overview and categorization of the state of the art in SP parallelization and elasticity, which is necessary to consolidate the state of the research and to plan future research directions on this basis. erefore, in this survey, we study the literature and develop a classi cation of current methods for both parallelization and elasticity in SP systems. or even di erent processing nodes in a shared-nothing cloud-based infrastructure. Frequent state synchronization must not hamper parallel processing, while the processing results have to remain consistent. Research proposes di erent approaches for parallel, stateless and stateful SP. ey di er in assumptions about the operator functions and state externalization mechanisms an SP system supports.is led to the development of a broad range of parallelization approaches tackling di erent problem cases.Second, how to continuously adapt the level of parallelization when the conditions of the SP operators, e.g. the workload or resources available, change at runtime. On the one hand, an SP system always needs enough resources to process the input data streams with a satisfying quality of service (QoS), e.g. latency or throughput. On the other hand, continuous provisioning of computing resources for peak workloads wastes resources at o -peak hours. us, an elastic SP system scales its resources according to the current need. Cloud computing provides on-demand resources to realize such elasticity [9]. e pay-as-you-go business model of cloud computing allows to cut costs by dynamically adapting the resource reservations to the needs of the SP system. It is challenging to strive the right balance between resource over-provisioning-which is costly, but is robust to workload uctuations-and on-demand scaling-which is cheap, but is vulnerable to sudden workload peaks. To this end, academia and industry developed elasticity methods. Again, they di er in their optimization objectives and assumptions about the operator parallelization model employed, the target system architecture, state management as well as timing and methodology.While there are many works that propose methods and solutions for speci c parallelization and elasticity problems in SP systems, there is a severe lack of overview, comparison, and classi cation of these methods. When we investigate...

show abstract

Section: Centralized Elasticitymentioning

confidence: 99%

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…Real-time stream processing system Previous work [Hidalgo et al 2017] has shown that autonomic stream processing systems are able to deal with burst of traffic that can be generated in the context of disaster scenarios, adjusting the internals of the systems to the current traffic. This process may enable our system to deal with the complexity of managing data analytics algorithms over environments subject to failures and integrate monitoring, planning, and execution capabilities so as to satisfy some utility goals (e.g., maximize performance, optimize resource usage, guarantees on processing reliability, etc.).…”

Section: Data Analytics In Iot and Disaster Scenariosmentioning

confidence: 99%

ADMITS: Architecting Distributed Monitoring and Analytics in IoT-based Disaster Scenarios

Pasquini¹,

Miani²,

Coelho³

et al. 2020

Anais Do Simpósio Brasileiro De Computação Ubíqua E Pervasiva (SBCUP 2020)

Self Cite

View full text Add to dashboard Cite

The ADMITS project aims to develop algorithms, protocols and architectures to enable a distributed computing environment to provide support for monitoring, failure detection, and analytics in IoT disaster scenarios. We face a context where, every year, millions of people are affected by natural and man-made disasters, whereby governments all around the world spend huge amounts of resources on preparation, immediate response, and reconstruction. Recently, the Internet of Things (IoT) paradigm has been extensively used for efficiently managing disaster scenarios, such as volcanic disasters, floods, forest fire, land- slides, earthquakes, urban disasters, industrial and terrorists attacks, and so on. However, in a disaster scenario the communication/processing infrastructure and the devices themselves may fail, producing either temporary or permanent network partitions and loss of information. Moreover, it is expected that in the years to come, IoT will generate large amounts of data, making processing and analysis challenging in time-critical applications. Considering such challenges, ADMITS targets the development of a architecture in which IoT, Fog, and Cloud computing technologies participate to provide required capabilities for IoT data analytics, real-time stream processing, and failure monitoring for environments potentially subject to disasters. In this positional paper, we discuss the motivation, objectives, architecture, research challenges (and how to overcome them) and initial efforts for the ADMITS project.

show abstract

“…A similar approach has been followed by [37] where the proposed methodology is able to control the number of replicas in streaming operators. The authors proposed two algorithms to be applied at different time-scales.…”

Section: Autoscaling Distributed Stream Processing Systemsmentioning

confidence: 99%

PASCAL: An architecture for proactive auto-scaling of distributed services

Lombardi

Muti

Aniello

et al. 2019

Future Generation Computer Systems

View full text Add to dashboard Cite

One of the main characteristics that today makes cloud services so popular is their ability to be elastic, i.e., they can adapt their provisioning to variable workloads, thus increasing resource utilization and reducing operating costs. At the core of any elastic service lies an automatic scaling mechanism that drives provisioning on the basis of a given strategy. In this paper we propose PASCAL, an architecture for Proactive Auto-SCALing of generic distributed services. PASCAL combines a proactive approach, to forecast incoming workloads, with a profiling system, to estimate required provision. Scale-in/out operations are decided according to an application-specific strategy, which aims at provisioning the minimum number of resources needed to sustain the foreseen workload. The main novelties introduced with PASCAL architecture are: (i) a strategy to proactively auto-scale a distributed stream processing system (namely, Apache Storm) with the aim of load balancing operators through an accurate system performance estimation model, and (ii) a strategy to proactively auto-scale a distributed datastore (namely, Apache Cassandra), focussed on how to choose when executing scaling actions on the basis of the time needed for the activation/deactivation of storage nodes so as to have the configuration ready when needed. We provide a prototype implementation of PASCAL for both use cases and, through an experimental evaluation conducted on a private cloud, we validate our approach and demonstrate the effectiveness of the proposed strategies in terms of saved resources and response time.

show abstract

Self-adaptive processing graph with operator fission for elastic stream processing

Cited by 44 publications

References 32 publications

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

ADMITS: Architecting Distributed Monitoring and Analytics in IoT-based Disaster Scenarios

PASCAL: An architecture for proactive auto-scaling of distributed services

Contact Info

Product

Resources

About