CoARC: Co-operative, Aggressive Recovery and Caching for Failures in Erasure Coded Hadoop

Subedi, Pradeep; Huang, Ping; Liu, Tong; Moore, Joseph; Skelton, Stan; He, Xubin

doi:10.1109/icpp.2016.40

Cited by 5 publications

(9 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The other approaches are: (1) Without a failure cache (No Cache), which is common in current big-data storage systems, as shown in Fig. 1; (2) FARC (failure ARC), which represents a simple adoption of the classic ARC [Megiddo and Modha (2003)]; (3) CoARC from related work [Subedi, Huang, Liu et al (2016)]. Note that, GFCache differs from FARC with our proposed greedy caching.…”

Section: Methodsmentioning

confidence: 99%

“…The most related work to our GFCache is CoARC from Subedi et al [Subedi, Huang and Liu (2016)], which features an LRF (least-recently-failed) failure caching algorithm and an aggressive recovery of all other temporarily unavailable blocks in the same stripe. Fig.…”

Section: Comparison With Coarcmentioning

confidence: 99%

“…Failure Caching In terms of failure caching in a setting of an erasure-coded storage system, very few research pays attention to the recurring data recovery problems. Sudedi et al [Subedi, Huang, Liu et al (2016)] treats per recovery process as a black box and first proposes CoARC, which essentially features a least-recently-failed (LRF) cache to buffering newly recovered data in order to eliminate repeated recoveries of the same data. This paper falls into the same track of reference [Subedi, Huang, Liu et al (2016)].…”

Section: Related Workmentioning

confidence: 99%

“…Therefore, the existing disposal of a recovered data would inevitably lead to repeated recoveries due to its recurring unavailability, thereby straining system resources and performance. Furthermore, statistics [Subedi, Huang, Liu et al (2016)] indicate that multiple-failure scenarios occur, and multiple blocks of a stripe can be unavailable simultaneously or incrementally. Given the current data recovering practice of one reconstruction operation per degraded read, multiple data corruptions on a stripe inevitably require multiple data recovery processes.…”

Section: Introductionmentioning

confidence: 99%

“…For instance, compared to the current system without a failure cache, GFCache manages to reduce the average latency of a request to 12.19%. Also, GFCache achieves 24.62% hit ratio than 22.74% of CoARC in Subedi et al [Subedi, Huang, Liu et al (2016)] under workload h3. The rest of the paper is organized as follows: Section II presents the background information.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

GFCache: A Greedy Failure Cache Considering Failure Recency and Failure Frequency for an Erasure-coded Storage System

Deng

Liu

Zhao

et al. 2019

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

In the big data era, data unavailability, either temporary or permanent, becomes a normal occurrence on a daily basis. Unlike the permanent data failure, which is fixed through a background job, temporarily unavailable data is recovered on-the-fly to serve the ongoing read request. However, those newly revived data is discarded after serving the request, due to the assumption that data experiencing temporary failures could come back alive later. Such disposal of failure data prevents the sharing of failure information among clients, and leads to many unnecessary data recovery processes, (e.g. caused by either recurring unavailability of a data or multiple data failures in one stripe), thereby straining system performance. To this end, this paper proposes GFCache to cache corrupted data for the dual purposes of failure information sharing and eliminating unnecessary data recovery processes. GFCache employs a greedy caching approach of opportunism to promote not only the failed data, but also sequential failure-likely data in the same stripe. Additionally, GFCache includes a FARC (Failure ARC) catch replacement algorithm, which features a balanced consideration of failure recency, frequency to accommodate data corruption with good hit ratio. The stored data in GFCache is able to support fast read of the normal data access. Furthermore, since GFCache is a generic failure cache, it can be used anywhere erasure coding is deployed with any specific coding schemes and parameters. Evaluations show that GFCache achieves good hit ratio with our sophisticated caching algorithm and manages to significantly boost system performance by reducing unnecessary data recoveries with vulnerable data in the cache.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Comparison With Coarcmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

GFCache: A Greedy Failure Cache Considering Failure Recency and Failure Frequency for an Erasure-coded Storage System

Deng

Liu

Zhao

et al. 2019

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

show abstract

PP: Popularity-based Proactive Data Recovery for HDFS RAID systems

Wei-dong

Mao

et al. 2018

Future Generation Computer Systems

View full text Add to dashboard Cite

Optimized Proactive Recovery in Erasure-Coded Cloud Storage Systems

et al. 2023

View full text Add to dashboard Cite

Cloud data centers have started utilizing erasure coding in large-scale storage systems to ensure high reliability with limited overhead compared to replication. However, data recovery in erasure coding incurs high network bandwidth consumption compared to replication. Cloud storage systems also play an important role in the energy consumption of data centers. Heuristic proactive recovery algorithms select all data blocks from failure-predicted disk/machine and perform proactive replication that contributes to huge recovery bandwidth savings. However, they fail to optimize the selection. Optimization can further improve resource savings. To address this issue, we propose a recovery algorithm that applies minimization on data blocks selected for proactive replication by considering the necessary and appropriate constraints that are constructed based on the system's current network traffic and data duplication information. We evaluate the proposed algorithm using extensive simulations. Experiments show that the recovery algorithm reduces network traffic by 60% and storage overhead by 46% compared to the heuristic proactive recovery approach. Also, the proposed proactive recovery methods reduce the storage system's energy consumption by up to 52% compared to replication.

show abstract

CoARC: Co-operative, Aggressive Recovery and Caching for Failures in Erasure Coded Hadoop

Cited by 5 publications

References 14 publications

GFCache: A Greedy Failure Cache Considering Failure Recency and Failure Frequency for an Erasure-coded Storage System

GFCache: A Greedy Failure Cache Considering Failure Recency and Failure Frequency for an Erasure-coded Storage System

PP: Popularity-based Proactive Data Recovery for HDFS RAID systems

Optimized Proactive Recovery in Erasure-Coded Cloud Storage Systems

Contact Info

Product

Resources

About