Analysis and implementation of software rejuvenation in cluster systems

Vaidyanathan, Kalyan; Harper, Richard E.; Hunter, S. W.; Trivedi, Kishor S.

doi:10.1145/378420.378434

Cited by 102 publications

(7 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Both these studies [19,8] found that nodes which just failed are more likely to fail again in the near future. At the same time, it has also been found [18] that software related error conditions can accumulate over time, leading to system failing in the long run.…”

Section: Related Workmentioning

confidence: 99%

Quantifying Temporal and Spatial Correlation of Failure Events for Proactive Management

2007

2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007)

View full text Add to dashboard Cite

Networked computing systems continue to grow in scale and in the complexity of their components and interactions. Component failures become norms instead of exceptions in these environments. Moreover, failure events exhibit strong correlations in time and space domain. In this paper, we develop a spherical covariance model with an adjustable timescale parameter to quantify the temporal correlation and a stochastic model to characterize spatial correlation. The models are further extended to take into account the information of application allocation to discover more correlations among failure instances. We cluster failure events based on their correlations and predict their future occurrences. Experimental results on a production coalition system, the Wayne State Grid, show the offline and online predictions by our predicting system can forecast 72.7% to 85.3% of the failure occurrences and capture failure correlations in cluster coalition environment.26th IEEE International Symposium on Reliable Distributed Systems

show abstract

Section: Related Workmentioning

confidence: 99%

Quantifying Temporal and Spatial Correlation of Failure Events for Proactive Management

2007

2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007)

View full text Add to dashboard Cite

show abstract

“…The introduced watchdog mechanism for fast software system recovery relates to the concept of software rejuvenation. Several works have investigated the phenomenon of "software aging" wherein the health of a software system degrades with time [38], [39]. These papers conclude that a mechanism which "rejuvenates" or "recovers" the software component to its stable state, would provide long-term benefits in terms of experienced system availability.…”

Section: Related Workmentioning

confidence: 99%

Response Time and Availability Study of RAFT Consensus in Distributed SDN Control Plane

Sakic

Kellerer

2018

IEEE Trans. Netw. Serv. Manage.

View full text Add to dashboard Cite

Software Defined Networking promises unprecedented flexibility and ease of network operations. While flexibility is an important factor when leveraging advantages of a new technology, critical infrastructure networks also have stringent requirements on network robustness and control plane delays. Robustness in the SDN control plane is realized by deploying multiple distributed controllers, formed into clusters for durability and fast-failover purposes. However, the effect of the controller clustering on the total system response time is not well investigated in current literature. Hence, in this work we provide a detailed analytical study of the distributed consensus algorithm RAFT, implemented in OpenDaylight and ONOS SDN controller platforms. In those controllers, RAFT implements the data-store replication, leader election after controller failures and controller state recovery on successful repairs. To evaluate its performance, we introduce a framework for numerical analysis of various SDN cluster organizations w.r.t. their response time and availability metrics. We use Stochastic Activity Networks for modeling the RAFT operations, failure injection and cluster recovery processes, and using real-world experiments, we collect the rate parameters to provide realistic inputs for a representative cluster recovery model. We also show how a fast rejuvenation mechanism for the treatment of failures induced by software errors can minimize the total response time experienced by the controller clients, while guaranteeing a higher system availability in the long-term.

show abstract

“…The original work [14] looked at telecommunication switches and long-running scientific programs. Another study examined rejuvenation in the context of clusters [22] and found timebased rejuvenation policies effective. However, their models assumed an increasing hazard function.…”

Section: Related Workmentioning

confidence: 99%

Improving cluster availability using workstation validation

Heath

Martin

Nguyen

2002

Proceedings of the 2002 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems

View full text Add to dashboard Cite

We demonstrate a framework for improving the availability of cluster based Internet services. Our approach models Internet services as a collection of interconnected components, each possessing well defined interfaces and failure semantics. Such a decomposition allows designers to engineer high availability based on an understanding of the interconnections and isolated fault behavior of each component, as opposed to ad-hoc methods. In this work, we focus on using the entire commodity workstation as a component because it possesses natural, fault-isolated interfaces. We define a failure event as a reboot because not only is a workstation unavailable during a reboot, but also because reboots are symptomatic of a larger class of failures, such as configuration and operator errors. Our observations of 3 distinct clusters show that the time between reboots is best modeled by a Weibull distribution with shape parameters of less than 1, implying that a workstation becomes more reliable the longer it has been operating. Leveraging this observed property, we design an allocation strategy which withholds recently rebooted workstations from active service, validating their stability before allowing them to return to service. We show via simulation that this policy leads to a 70-30 rule-of-thumb: For a constant utilization, approximately 70% of the workstation failures can be masked from end clients with 30% extra capacity added to the cluster, provided reboots are not strongly correlated. We also found our technique is most sensitive to the burstiness of reboots as opposed to absolute lengths of workstation uptimes.½ Of course, cluster-based systems include other important components such as network-attached disks, hubs, switches, and cables. Characterizing these components is beyond the scope of the immediate paper but is clearly important to the overall application of the framework to the design of Internet services.¾ In actuality, a better metric is Time-To-Failure (TTF): that is, the time from when a machine starts until it reaches some definition of failure (e.g., a clean shutdown or crash). Unfortunately, the system logs did not contain sufficient information to compute TTFs. Thus, we were forced to approximate TTF with TTB.

show abstract

Analysis and implementation of software rejuvenation in cluster systems

Cited by 102 publications

References 24 publications

Quantifying Temporal and Spatial Correlation of Failure Events for Proactive Management

Quantifying Temporal and Spatial Correlation of Failure Events for Proactive Management

Response Time and Availability Study of RAFT Consensus in Distributed SDN Control Plane

Improving cluster availability using workstation validation

Contact Info

Product

Resources

About