Hard Drive Failure Prediction Using Big Data

Yang, Wenjun; Hu, Dianming; Liu, Yuliang; Wang, Shuhao; Jiang, Tianming

doi:10.1109/srdsw.2015.15

Cited by 31 publications

(8 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The time to failure or censoring of each hard drive and its related smart stats values are obtained using SQL queries. According to Yang et al 26 . and Klein 27 smart stats given in Table 6 can be considered informational:…”

Section: Fitting the Backblaze Hard Drive Datamentioning

confidence: 99%

See 1 more Smart Citation

The Weibull log‐logistic mixture distributions: Model, theory and application to lifetime data

Azzaz

Boudrissa

2020

Quality & Reliability Eng

View full text Add to dashboard Cite

Lifetime data collected from reliability tests are among data that often exhibit significant heterogeneity caused by variations in manufacturing, which makes standard lifetime models inadequate. Finite mixture models provide more flexibility for modeling such data. In this paper, the Weibull‐log‐logistic mixture distributions model is introduced as a new class of flexible models for heterogeneous lifetime data. Some statistical properties of the model are presented including the failure rate function, moments generating function, and characteristic function. The identifiability property of the class of all finite mixtures of Weibull‐log‐logistic distributions is proved. The maximum likelihood estimation (MLE) of model parameters under the Type I and Type II censoring schemes is derived. Some numerical illustrations are performed to study the behavior of the obtained estimators. The model is applied to the hard drive failure data made by the Backblaze data center, where it is found that the proposed model provides more flexibility than the univariate life distributions (Weibull, Exponential, logistic, log‐logistic, Frechet). The failure rate of hard disk drives (HDDs) is obtained based on MLE estimates. The analysis of the failure rate function on the basis of SMART attributes shows that the failure of HDDs can have different causes and mechanisms.

show abstract

Section: Fitting the Backblaze Hard Drive Datamentioning

confidence: 99%

“…The logistic regression is performed assuming a Gompertz link function, which provides more flexibility than Logit and Probit functions (Trexler et al., 34 XLSTAT 33 ). To optimize the logistic regression, smart stats are discretized, assuming equal frequencies (Yang et al 26 …”

Section: Fitting the Backblaze Hard Drive Datamentioning

confidence: 99%

The Weibull log‐logistic mixture distributions: Model, theory and application to lifetime data

Azzaz

Boudrissa

2020

Quality & Reliability Eng

View full text Add to dashboard Cite

show abstract

“…Through the years, various ML and reasoning approaches based on algorithms such as Decision Trees (DT) [ 8 ], k-Nearest Neighbour [ 9 ], Classification Trees (CT) [ 10 ], and Regression Trees (RT) [ 11 ] have been adopted successfully for PdM (i.e., the failure prediction accuracy rates have improved) [ 12 , 13 , 14 ]. State of the art solutions such as [ 7 , 15 , 16 , 17 , 18 ] have shown to achieve more than 90% accuracy in failure prediction. However, ML on its own lacks context awareness [ 19 ] and therefore, lacks the benefits that comes with it.…”

Section: Introductionmentioning

confidence: 99%

Knowledge Graph Based Hard Drive Failure Prediction

Chhetri

Kurteva

Adigun

et al. 2022

Sensors

View full text Add to dashboard Cite

The hard drive is one of the important components of a computing system, and its failure can lead to both system failure and data loss. Therefore, the reliability of a hard drive is very important. Realising this importance, a number of studies have been conducted and many are still ongoing to improve hard drive failure prediction. Most of those studies rely solely on machine learning, and a few others on semantic technology. The studies based on machine learning, despite promising results, lack context-awareness such as how failures are related or what other factors, such as humidity, influence the failure of hard drives. Semantic technology, on the other hand, by means of ontologies and knowledge graphs (KGs), is able to provide the context-awareness that machine learning-based studies lack. However, the studies based on semantic technology lack the advantages of machine learning, such as the ability to learn a pattern and make predictions based on learned patterns. Therefore, in this paper, leveraging the benefits of both machine learning (ML) and semantic technology, we present our study, knowledge graph-based hard drive failure prediction. The experimental results demonstrate that our proposed method achieves higher accuracy in comparison to the current state of the art.

show abstract

“…This can cost in the form of server down, erased backup, lower reliability, unavailability of Internet and fiasco to fetch the latest data and compromised data storage in the data centers. To minimize the problem, HDDs conditions are monitored which helps in the detection of soon to fail drives [Strom, Lee, Tyndall et al (2007); Ma, Traylor, Douglis et al (2015); Yang, Hu, Liu et al (2015)]. The conditions of HDDs are supervised using sensors like acoustic emission, accelerometers, counters and thermal sensors.…”

Section: Introductionmentioning

confidence: 99%

Failure Prediction, Lead Time Estimation and Health Degree Assessment for Hard Disk Drives Using Voting based Decision Trees

Kaur¹,

Kaur²

2019

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Hard Disk drives (HDDs) are an essential component of cloud computing and big data, responsible for storing humongous volumes of collected data. However, HDD failures pose a huge challenge to big data servers and cloud service providers. Every year, about 10% disk drives used in servers crash at least twice, lead to data loss, recovery cost and lower reliability. Recently, the researchers have used SMART parameters to develop various prediction techniques, however, these methods need to be improved for reliability and real-world usage due to the following factors: they lack the ability to consider the gradual change/deterioration of HDDs; they have failed to handle data unbalancing and biases problem; they don't have adequate mechanisms for health status prediction of HDDs. This paper introduces a novel voting-based decision tree classifier to cater failure prediction, a balance splitting algorithm for the data unbalancing problem, an advanced procedure for lead time estimation and R-CNN based approach for health status estimation. Our system works robustly by considering a gradual change in SMART parameters. The system is rigorously tested on 3 datasets and it delivered benchmarks results as compared to the state of the art.

show abstract

Hard Drive Failure Prediction Using Big Data

Cited by 31 publications

References 6 publications

The Weibull log‐logistic mixture distributions: Model, theory and application to lifetime data

The Weibull log‐logistic mixture distributions: Model, theory and application to lifetime data

Knowledge Graph Based Hard Drive Failure Prediction

Failure Prediction, Lead Time Estimation and Health Degree Assessment for Hard Disk Drives Using Voting based Decision Trees

Contact Info

Product

Resources

About