An optimal checkpointing model with online OCI adjustment for stream processing applications

Zhuang, Yuan; Wei, Xiaohui; Li, Hongliang; Wang, Yongfang; He, Xubin

doi:10.1002/cpe.5347

Cited by 5 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…TA B L E 1 Factors considered in existing work proposed to determine the optimal checkpoint interval Authors Factors Young 10 Daly firstorder 20 Daly 11 Jin et al 46 Naksinehab-oon et al 22 Ling et al 23 Zhuang et al 47 Liu et al 48 Shastry et al 49 Zhu et al 50 Failure during computation…”

Section: Checkpoint Interval Optimizationsmentioning

confidence: 99%

Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

2021

View full text Add to dashboard Cite

Fault-tolerance is an essential part of a stream processing system that guarantees data analysis could continue even after failures. State-of-the-art distributed stream processing systems use checkpointing to support fault-tolerance for stateful computations where the state of the computations is periodically persisted. However, the frequency of performing checkpoints impacts the performance (utilization, latency, and throughput) of the system as the checkpointing process consumes resources and time that can be used for actual computations. In practice, systems are often configured to perform checkpoints based on crude values ignoring factors such as checkpoint and restart costs, leading to suboptimal performance. In our previous work, we proposed a theoretical optimal checkpoint interval that maximizes the system utilization for stream processing systems to minimize the impact of checkpointing on system performance.In this article, we investigate the practical benefits of our proposed theoretical optimal by conducting experiments in a real-world cloud setting using different streaming applications; we use Apache Flink, a well-known stream processing system for our experiments. The experiment results demonstrate that an optimal interval can achieve better utilization, confirming the practicality of the theoretical model when applied to real-world applications. We observed utilization improvements from 10% to 200% for a range of failure rates from 0.3 failures per hour to 0.075 failures per minute. Moreover, we explore how performance measures: latency and throughput are affected by the optimal interval.Our observations demonstrate that significant improvements can be achieved using the optimal interval for both latency and throughput.

show abstract

Section: Checkpoint Interval Optimizationsmentioning

confidence: 99%

Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

2021

View full text Add to dashboard Cite

show abstract

“…For the general BPP and the two-dimensional BPP (2BP) problem in particular, please refer to surveys. [27][28][29][30] Considering the assigned processor number (parallelism degree) of a component as the first dimension of the BPP, and the execution time of a component as the second dimension, the process scheduling problem is similar to two-dimensional packing problems, such as the 2BP and two-dimensional strip packing problem (2SP). 31 Both problems are strongly NP-hard.…”

Section: Two-dimensional Strip Packingmentioning

confidence: 99%

Coordinated process scheduling algorithms for coupled earth system models

Wei

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

It is becoming increasingly significant for humans to predict and understand future climate changes using coupled climate system models. Although the performance and scalability of individual physical components have improved over the past few years, coupled climate systems still suffer from low efficiency. This paper focuses on the process scheduling problem for the widely applied coupled earth system model (CESM). The proposed resource allocation strategies allow components to execute on a compromised suboptimal setup and still maintain approximately the best parallel speedup. With this flexible resource allocation strategy, we further propose a coordinated process scheduling algorithm (CPSA). More notably, we propose an upgraded version called CPSA-B, which makes efficient resource sharing configurations, including resource allocation and process layout of components. We integrate CPSA and CPSA-B as pre-arrangement tools into the CESM program and deploy them on the Huawei Kunpeng platform. The speedup curves of the CESM components are prepared in advance, based on sampling tests. Experimental data show that CPSA-B reduces up to 58% of the execution time compared with the CESM default strategy. The algorithm has low complexity and can efficiently find solutions for large input sizes. K E Y W O R D S coupled climate system model, process scheduling, resource allocation, strip packing INTRODUCTIONNowadays, changes of climate have become a critical subject in that they have potentially devastating impact on both our economy and society.Analysis and prediction of future global or regional climate are undoubtedly vital. 1,2 However, climate simulation is a very challenging topic because it involves a large number of resource-consuming physical procedures. 3 In recent years, state-of-the-art climate system models have been developedto integrate observation data with simulation programs of the Earth's climate system, including the atmosphere, ocean, and land surface. 4Coupled climate system models are usually complex and extremely resource-consuming. The National Center for Atmospheric Research reported that an improvement in each percent of the climate simulations in time-to-solution performance frees up to the equivalent of $250,000 in computing resources. 5 Besides, with increasing demands of a larger solution space and extension of simulation grid, these climate models are becoming even more resource consuming.One of these coupled climate systems, the community earth system model (CESM) is a widely applied global climate model. It is a fully coupled system of multigeophysical components. The cyber-physics procedures include the atmosphere component (ATM), 6 ocean component (OCN), 7sea ice component (ICE), 8 and land surface component (LND). 9 The CESM framework executes all these components in an iteration, calculating the results for one simulation day. At the beginning of each iteration, these components periodically exchange model boundaries with a coupler program (CPL). 10

show abstract

“…Most works concern checkpointing and snapshot techniques. Zhuang et al [11] propose a novel Optimal Checkpointing Model for stream processing. This model proposes a dynamic calculation of an optimal checkpointing interval, aiming to obtain an optimal processing efficiency.…”

Section: A Reliability and Fault Tolerancementioning

confidence: 99%

On the Cost of Acking in Data Stream Processing Systems

Pagliari

Huet

Urvoy-Keller

2019

2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

View full text Add to dashboard Cite

The widespread use of social networks and applications such as IoT networks generates a continuous stream of data that companies and researchers want to process, ideally in real-time. Data stream processing systems (DSP) enable such continuous data analysis by implementing the set of operations to be performed on the stream as directed acyclic graph (DAG) of tasks. While these DSP systems embed mechanisms to ensure fault tolerance and message reliability, only few studies focus on the impact of these mechanisms on the performance of applications at runtime. In this paper, we demonstrate the impact of the message reliability mechanism on the performance of the application. We use an experimental approach, using the Storm middleware, to study an acknowledgment-based framework. We compare the two standard schedulers available in Storm with applications of various degrees of parallelism, over single and multi cluster scenarios. We show that the acking layer may create an unforeseen bottleneck due to the acking tasks placement; a problem which, to the best of our knowledge, has been overlooked in the scientific and technical literature. We propose two strategies for improving the acking tasks placement and demonstrate their benefit in terms of throughput and latency.

show abstract

An optimal checkpointing model with online OCI adjustment for stream processing applications

Cited by 5 publications

References 31 publications

Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

Coordinated process scheduling algorithms for coupled earth system models

On the Cost of Acking in Data Stream Processing Systems

Contact Info

Product

Resources

About