Predictive Models of Hard Drive Failures Based on Operational Data

Aussel, Nicolas; Jaulin, Samuel; Gandon, Guillaume; Petetin, Yohan; Fazli, Eriza Hafid; Chabridon, Sophie

doi:10.1109/icmla.2017.00-92

Cited by 34 publications

(21 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many researchers and industrial partners have also focused their efforts on analyzing real-world datasets. For example, there are plenty of works in the literature that concentrated on using Backblaze's public dataset to extract some useful models [21]- [28]. Other works such as [29] base their analysis (latent sector error analysis for reliability in this case) on proprietary data collected from production-ready storage systems.…”

Section: A Related Workmentioning

confidence: 99%

“…A case study for predicting hard disk's time-to-failure using regression analysis is given by [24]. The authors in [21] have used various machine learning classification methods to predict hard drive failures. On the other hand, [22] studies statistical techniques to predict disk replacement with 98% accuracy using 30, 000 drives of Backblaze's dataset.…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

On the Distribution Modeling of Heavy-Tailed Disk Failure Lifetime in Big Data Centers

Arslan

Zeydan

2021

IEEE Trans. Rel.

View full text Add to dashboard Cite

It has become commonplace to observe frequent multiple disk failures in big data centers in which thousands of drives operate simultaneously. Disks are typically protected by replication or erasure coding to guarantee a predetermined reliability. However, in order to optimize data protection, real life disk failure trends need to be modeled appropriately. The classical approach to modeling is to estimate the probability density function of failures using non-parametric estimation techniques such as Kernel Density Estimation (KDE). However, these techniques are suboptimal in the absence of the true underlying density function. Moreover, insufficient data may lead to overfitting. In this study, we propose to use a set of transformations to the collected failure data for almost perfect regression in the transform domain. Then, by inverse transformation, we analytically estimated the failure density through the efficient computation of moment generating functions and hence the density functions. Moreover, we developed a visualization platform to extract useful statistical information such as model-based mean time to failure. Our results indicate that for other heavy-tailed data, complex Gaussian Hypergeometric Distribution (GHD) and classical KDE approach can perform best if overfitting problem can be avoided and complexity burden is overtaken. On the other hand, we show that the failure distribution exhibits less complex Argus-like distribution after performing Box-Cox transformation up to appropriate scaling and shifting operations.

show abstract

Section: A Related Workmentioning

confidence: 99%

Section: A Related Workmentioning

confidence: 99%

On the Distribution Modeling of Heavy-Tailed Disk Failure Lifetime in Big Data Centers

Arslan

Zeydan

2021

IEEE Trans. Rel.

View full text Add to dashboard Cite

show abstract

“…In [10], the authors came up with disk replacement prediction algorithm with changepoint detection in time series Backblaze data and concluded some rules for directly identifying the state of a device: healthy or faulty. Aussel et al, [6] used the same dataset to perform hard drive failure prediction with SVM, RF and GBT and discussed their performances based on precision and recall. Prediction of remaining useful lives using quantum particle swarm optimization [11] [12] of lithium-ion battery has been discussed in [13] and a host of recent swarm intelligence algorithms [14] can be effectively applied in prediction of RUL of various devices in conjuction with other ML approaches.…”

Section: B Related Workmentioning

confidence: 99%

“…But as this does not indicate the actual efficiency of the prediction model, this cannot be used in a real life scenario. In [6] the authors claimed that the best performance on Backblaze dataset was shown by Random Forest (RF). The Precision and Recall values based on the threshold of device failure within 10 days were recorded as almost 0.93 and 0.6 dividing the dataset into training and test sets using cross-validation techniques, whereas we obtained an average precision of 0.84 and recall of 0.72 using the decision threshold of device failure within 10 days without using any future information in the simulation process.…”

Section: Comparative Analyses With Existing Researchmentioning

confidence: 99%

“…To make things worse, at some of the feature values deemed healthy for one device, another device may fail. It should be noted that the data has some failure states which have similar feature sets to those of active states [6]. This renders the possibility of finding out one specific set of feature values corresponding to global failure slim.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Mechanisms for Integrated Feature Normalization and Remaining Useful Life Estimation Using LSTMs Applied to Hard-Disks

Basak

Sengupta

Dubey

2019

2019 IEEE International Conference on Smart Computing (SMARTCOMP)

View full text Add to dashboard Cite

In this paper we focus on application of data-driven methods for remaining useful life estimation in components where past failure data is not uniform across devices, i.e. there is a high variance in the minimum and maximum value of the key parameters. The system under study is the hard disks used in computing cluster. The data used for analysis is provided by Backblaze as discussed later. In the article, we discuss the architecture of of the long short term neural network used and describe the mechanisms to choose the various hyper-parameters. Further, we describe the challenges faced in extracting effective training sets from highly unorganized and class-imbalanced big data and establish methods for online predictions with extensive data pre-processing, feature extraction and validation through online simulation sets with unknown remaining useful lives of the hard disks. Our algorithm performs especially well in predicting RUL near the critical zone of a device approaching failure. With the proposed approach we are able to predict whether a disk is going to fail in next ten days with an average precision of 0.8435. We also show that the architecture trained on a particular model is generalizable and transferable as it can be used to predict RUL for devices in other models from same manufacturer.

show abstract