ChronoStream: Elastic stateful stream computation in the cloud

Wu, Yingjun; Tan, Kian-Lee

doi:10.1109/icde.2015.7113328

Cited by 72 publications

(67 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Solutions for the domain of continuous SQL-style queries can be found in [21]. For stateful UDFs, [10], [12], [14] demonstrate solutions to the management of key-partitioned task state in the face of elastic scaling.…”

Section: A Assumptionsmentioning

confidence: 99%

Elastic Stream Processing with Latency Guarantees

Lohrmann

Janácik

Kao

2015

2015 IEEE 35th International Conference on Distributed Computing Systems

105

View full text Add to dashboard Cite

Many Big Data applications in science and industry have arisen, that require large amounts of streamed or event data to be analyzed with low latency. This paper presents a reactive strategy to enforce latency guarantees in data flows running on scalable Stream Processing Engines (SPEs), while minimizing resource consumption. We introduce a model for estimating the latency of a data flow, when the degrees of parallelism of the tasks within are changed. We describe how to continuously measure the necessary performance metrics for the model, and how it can be used to enforce latency guarantees, by determining appropriate scaling actions at runtime. Therefore, it leverages the elasticity inherent to common cloud technology and cluster resource management systems. We have implemented our strategy as part of the Nephele SPE. To showcase the effectiveness of our approach, we provide an experimental evaluation on a large commodity cluster, using both a synthetic workload as well as an application performing real-time sentiment analysis on real-world social media data.

show abstract

Section: A Assumptionsmentioning

confidence: 99%

Elastic Stream Processing with Latency Guarantees

Lohrmann

Janácik

Kao

2015

2015 IEEE 35th International Conference on Distributed Computing Systems

105

View full text Add to dashboard Cite

show abstract

“…There are multiple PSPEs [12,18,35,40] that support horizontal scaling and load balancing, by collecting system-wide statistics over a certain timespan, then making global decisions on the basis of these. However, these techniques do not consider collocation of operator instances to reduce communication overhead.…”

Section: Dynamic Scalingmentioning

confidence: 99%

“…The state migration cost is calculated by the cost model given in the referenced paper. All the solutions proposed in this paper, are independent on the actual state migration technique applied, hence alternative state migration techniques [9,27,40] can be applied.…”

Section: System Modelmentioning

confidence: 99%

Integrative Dynamic Reconfiguration in a Parallel Stream Processing Engine

Madsen

Zhou

Cao

2017

2017 IEEE 33rd International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

Load balancing, operator instance collocations and horizontal scaling are critical issues in Parallel Stream Processing Engines to achieve low data processing latency, optimized cluster utilization and minimized communication cost respectively. In previous work, these issues are typically tackled separately and independently. We argue that these problems are tightly coupled in the sense that they all need to determine the allocations of workloads and migrate computational states at runtime. Optimizing them independently would result in suboptimal solutions. Therefore, in this paper, we investigate how these three issues can be modeled as one integrated optimization problem. In particular, we first consider jobs where workload allocations have little effect on the communication cost, and model the problem of load balance as a Mixed-Integer Linear Program. Afterwards, we present an extended solution called ALBIC, which support general jobs. We implement the proposed techniques on top of Apache Storm, an open-source Parallel Stream Processing Engine. The extensive experimental results over both synthetic and real datasets show that our techniques clearly outperform existing approaches.

show abstract

“…As application state is often kept by multiple operators, fault tolerance is achieved either by replicating sections of the processing graph or via upstream backup. The former demands synchronisation of operators via a protocol such as Flux [66] or other transactional protocols [67], whereas the latter, when a node fails, requires parents to replay previously sent messages to rebuild the state.…”

Section: Batch Operation Inputmentioning

confidence: 99%

“…Wu and Tan [67] discuss technical challenges that may require a redesign of distributed stream processing systems, such as maintaining large amounts of state, workload fluctuation and multi-tenant resource sharing. They introduce ChronoStream, a system to support elasticity and high availability in latency-sensitive stream computing.…”

Section: Online Techniquesmentioning

confidence: 99%

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

Assuno

Veith

Buyya

2018

Journal of Network and Computer Applications

257

View full text Add to dashboard Cite

Under several emerging application scenarios, such as in smart cities, operational monitoring of large infrastructure, wearable assistance, and Internet of Things, continuous data streams must be processed under very short delays. Several solutions, including multiple software engines, have been developed for processing unbounded data streams in a scalable and efficient manner. More recently, architecture has been proposed to use edge computing for data stream processing. This paper surveys state of the art on stream processing engines and mechanisms for exploiting resource elasticity features of cloud computing in stream processing. Resource elasticity allows for an application or service to scale out/in according to fluctuating demands. Although such features have been extensively investigated for enterprise applications, stream processing poses challenges on achieving elastic systems that can make efficient resource management decisions based on current load. Elasticity becomes even more challenging in highly distributed environments comprising edge and cloud computing resources. This work examines some of these challenges and discusses solutions proposed in the literature to address them.

show abstract

ChronoStream: Elastic stateful stream computation in the cloud

Cited by 72 publications

References 22 publications

Elastic Stream Processing with Latency Guarantees

Elastic Stream Processing with Latency Guarantees

Integrative Dynamic Reconfiguration in a Parallel Stream Processing Engine

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

Contact Info

Product

Resources

About