Cloud²HDD: Large-Scale HDD Data Analysis on Cloud for Cloud Datacenters

Abstract

The main focus of this paper is to develop a distributed large scale data analysis platform for the opensource data of Backblaze cloud datacenter which consists of operational hard disk drive (HDD) information collected over an observable period of 2272 days (over 74 months). To carefully analyze the intrinsic characteristics of the hard disk behavior, we have exploited a large bolume of data and the benefits of Hadoop ecosystem as our big data processing engine. In other words, we have utilized a special distributed scheme on cloud for cloud HDD data, which is termed as Cloud 2 HDD. To classify the remaining lifetime of hard disk drives based on health indicators such as in-built S.M.A.R.T (Self-Monitoring, Analysis, and Reporting Technology) features, we used some of the state-of-the-art classification algorithms and compared their accuracy, precision, and recall rates simultaneously. In addition, importance of various S.M.A.R.T. features in predicting the true remaining lifetime of HDDs are identified. For instance, our analysis results indicate that Random Forest Classifier (RFC) can yield up to 94% accuracy with the highest precision and recall at a reasonable time by classifying the remaining lifetime of drives into one of three different classes, namely critical, high and low ideal states in comparison to other classification approaches based on a specific subset of S.M.A.R.T. features.

Durability and Availability of Erasure-Coded Storage Systems with Concurrent Maintenance

“…The heavy-tailedness of disk failure lifetime is quite well known and modeled in literature [8]. One of the most critical trend is to use collected data (For instance, Backblaze provides such datasets for research [9]) to model data storage system reliability for predictive maintenance [10]. Later in the document (Section 6) we will demonstrate one specific use case of data for modeling complex systems to help with the conventional mathematical tools for estimating dependibility of storage systems.…”

“…Suppose further that x of these requests are from the failed set, and k −x are from the available and operational ones. Due to sampling without replacement, probability of that happening is given by the hypergeometric distribution 10 . In this particular condition, we need to wait for the x failed carriers to be repaired first which is given by the maximum repair time and typically not distributed exponentially.…”

“…The present discussion only slightly changes in case such class of codes are used instead. 10 Sampling with replacement would lead to a Binomially distributed statistics instead.…”

On the Distribution Modeling of Heavy-Tailed Disk Failure Lifetime in Big Data Centers

“…Many researchers and industrial partners have also focused their efforts on analyzing real-world datasets. For example, there are plenty of works in the literature that concentrated on using Backblaze's public dataset to extract some useful models [21]- [28]. Other works such as [29] base their analysis (latent sector error analysis for reliability in this case) on proprietary data collected from production-ready storage systems.…”