What Can We Learn from Four Years of Data Center Hardware Failures?

Wang, Guosai; Zhang, Lifei; Xu, Wei

doi:10.1109/dsn.2017.26

Cited by 94 publications

(43 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several studies describe and analyze failures in large scale computing systems using data coming from production systems [14], [15], [17], [25], [27], [29]. Other work focus on the failures related to specific hardware components (e.g., hard drives [21], [24], GPUs [20], or memory [12], [28]).…”

Section: A Analysis Of Failures In Large Scale Systemsmentioning

confidence: 99%

See 1 more Smart Citation

CPU Overheating Characterization in HPC Systems: A Case Study

Platini

Ropars

Pelletier

et al. 2018

2018 IEEE/ACM 8th Workshop on Fault Tolerance for HPC at eXtreme Scale (FTXS)

View full text Add to dashboard Cite

With the increase in size of supercomputers, also increases the number of abnormal events. Some of these events might lead to an application failure. Others might simply impact the system efficiency. CPU overheating is one such event that decreases the system efficiency: when a CPU overheats, it reduces its frequency. This paper studies the problem of CPU overheating in supercomputers. In a first part, we analyze data collected over one year on a supercomputer of the top500 list to understand under which conditions CPU overheating occurs. Our analysis show that overheating events are due to some specific applications. In a second part, we evaluate the impact of such overheating events on the performance of MPI applications. Using 6 representative HPC benchmarks, we show that for a majority of the applications, a frequency drop on one CPU impacts the execution time of distributed runs proportionally to the duration and to the extent of the frequency drop.

show abstract

Section: A Analysis Of Failures In Large Scale Systemsmentioning

confidence: 99%

“…Abnormal events in supercomputers that lead to crash failures [15], [17], [25], [29] or corrupted states [12], [28] are widely studied in the literature. On the other hand, a CPU overheating is an event that limits the performance of the system without directly compromising the results of the executions.…”

Section: Introductionmentioning

confidence: 99%

CPU Overheating Characterization in HPC Systems: A Case Study

Platini

Ropars

Pelletier

et al. 2018

2018 IEEE/ACM 8th Workshop on Fault Tolerance for HPC at eXtreme Scale (FTXS)

View full text Add to dashboard Cite

show abstract

“…B. Schroeder et al [3], provided a quantitative analysis of hard disk replacement rates and discussed the statistical properties of the distribution capturing time between replacements. On the other hand, Wang et al [4], concludes that the time between failure cannot be captured through any of the standard distributions and commented on the difficulty of capturing fault trend using standard distributions.…”

Section: B Related Workmentioning

confidence: 99%

Mechanisms for Integrated Feature Normalization and Remaining Useful Life Estimation Using LSTMs Applied to Hard-Disks

Basak

Sengupta

Dubey

2019

2019 IEEE International Conference on Smart Computing (SMARTCOMP)

View full text Add to dashboard Cite

In this paper we focus on application of data-driven methods for remaining useful life estimation in components where past failure data is not uniform across devices, i.e. there is a high variance in the minimum and maximum value of the key parameters. The system under study is the hard disks used in computing cluster. The data used for analysis is provided by Backblaze as discussed later. In the article, we discuss the architecture of of the long short term neural network used and describe the mechanisms to choose the various hyper-parameters. Further, we describe the challenges faced in extracting effective training sets from highly unorganized and class-imbalanced big data and establish methods for online predictions with extensive data pre-processing, feature extraction and validation through online simulation sets with unknown remaining useful lives of the hard disks. Our algorithm performs especially well in predicting RUL near the critical zone of a device approaching failure. With the proposed approach we are able to predict whether a disk is going to fail in next ten days with an average precision of 0.8435. We also show that the architecture trained on a particular model is generalizable and transferable as it can be used to predict RUL for devices in other models from same manufacturer.

show abstract

“…is the standard score of x, yi−ȳ sy is the standard score of y, x and y are two datasets containing n values of a pair of resource use counters or a pair of events, 2 is the sample standard deviation of y,x andȳ is the sample mean of x and y.…”

Section: Correlationmentioning

confidence: 99%

“…This research is summarized briefly below and analyzed in Section V. HPC and data center operators face challenges on a day by day basis. Recent work that studied HPC and data center failures [1], [2] have shown that the mean time between failures is decreasing and it is important for the system administrators to identify errors quickly. However, these high-performance systems generate a massive amount of data.…”

Section: Introductionmentioning

confidence: 99%

Enabling Dependability-Driven Resource Use and Message Log-Analysis for Cluster System Diagnosis

Chuah

Jhumka

Alt

et al. 2017

2017 IEEE 24th International Conference on High Performance Computing (HiPC)

View full text Add to dashboard Cite

Recent work have used both failure logs and resource use data separately (and together) to detect system failureinducing errors and to diagnose system failures. System failure occurs as a result of error propagation and the (unsuccessful) execution of error recovery mechanisms. Knowledge of error propagation patterns and unsuccessful error recovery is important for more accurate and detailed failure diagnosis, and knowledge of recovery protocols deployment is important for improving system reliability. This paper presents the CORRMEXT framework which carries failure diagnosis another significant step forward by analyzing and reporting error propagation patterns and degrees of success and failure of error recovery protocols. CORRMEXT uses both error messages and resource use data in its analyses. Application of CORRMEXT to data from the Ranger supercomputer have produced new insights. CORRMEXT has: (i) identified correlations between resource use counters that capture recovery attempts after an error, (ii) identified correlations between error events to capture error propagation patterns within the system, (iii) identified error propagation and recovery paths during system execution to explain system behaviour, (iv) showed that the earliest times of change in system behaviour can only be identified by analyzing both the correlated resource use counters and correlated errors. CORRMEXT will be installed on the HPC clusters at the Texas Advanced Computing Center in Autumn 2017.

show abstract

What Can We Learn from Four Years of Data Center Hardware Failures?

Cited by 94 publications

References 22 publications

CPU Overheating Characterization in HPC Systems: A Case Study

CPU Overheating Characterization in HPC Systems: A Case Study

Mechanisms for Integrated Feature Normalization and Remaining Useful Life Estimation Using LSTMs Applied to Hard-Disks

Enabling Dependability-Driven Resource Use and Message Log-Analysis for Cluster System Diagnosis

Contact Info

Product

Resources

About