Analyzing a Five-Year Failure Record of a Leadership-Class Supercomputer

Abstract

Extreme-scale computing systems are required to solve some of the grand challenges in science and technology. From astrophysics to molecular biology, supercomputers are an essential tool to accelerate scientific discovery. However, large computing systems are prone to failures due to their complexity. It is crucial to develop an understanding of how these systems fail to design reliable supercomputing platforms for the future. This paper examines a five-year failure and workload record of a leadership-class supercomputer. To the best of our knowledge, five years represents the vast majority of the lifespan of a supercomputer. This is the first time such analysis is performed on a top 10 modern supercomputer. We performed a failure categorization and found out that: i) most errors are GPUrelated, with roughly 37% of them being double-bit errors on the cards; ii) failures are not evenly spread across the physical machine, with room temperature presumably playing a major role; and iii) software errors of the system bring down several nodes concurrently. Our failure rate analysis unveils that: i) the system consistently degrades, being at least twice as reliable at the beginning, compared to the end of the period; ii) Weibull distribution closely fits the meantime between failure data; and iii) hardware and software errors show a markedly different pattern. Finally, we correlated failure and workload records to reveal that: i) failure and workload records are weakly correlated, except for certain types of failures when segmented by the hours of the day; ii) several categories of failures make jobs crash within the first minutes of execution; and iii) a significant fraction of failed jobs exhaust the requested time with a disregard of when the failure occurred during execution.

An unsupervised machine-learning checkpoint-restart algorithm using Gaussian mixtures for particle-in-cell simulations

“…The very large total number of components will lead to frequent failures, even though the mean time between failures (MTBF) for the individual components may be large. For instance, while the MTBF of a CPU can be months to years [2], that of current supercomputers can be within a few hours [3,4]. With billion-core parallelism at exascale, the MTBF has been projected to be within (or even far below) one hour [5,6].…”

Design and Study of Elastic Recovery in HPC Applications

“…Recent studies show that modern HPC systems have several failures per day [16], [32], consequently, we set MTBF in equations 5 and 6 equal to 6 hours. This choice corresponds to the estimated MTBF when the application uses 15K cores [9].…”

An unsupervised machine-learning checkpoint-restart algorithm using Gaussian mixtures for particle-in-cell simulations

“…The very large total number of components will lead to frequent failures, even though the mean time between failures (MTBF) for the individual components may be large. For instance, while the MTBF of a CPU can be months to years [2], that of current supercomputers can be a few hours or less [3], [4]. With billion-core parallelism at exascale, the MTBF has been projected to be within (or even significantly below) one hour [5], [6].…”