Convertible Codes: Enabling Efficient Conversion of Coded Data in Distributed Storage

Maturana, Francisco; Rashmi, K. V.

doi:10.1109/tit.2022.3155972

Cited by 10 publications

(35 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [66], the authors propose two erasure codes designed to undergo a specific transition in parameters. In [33], the authors propose a general theoretical framework for studying codes that enable efficient transitions for general parameters, and derive lower bounds on the cost of transitions as well as describe optimal code constructions for certain specific parameters. However, none of the existing code constructions are applicable for the diverse set of transitions needed for disk-adaptive redundancy in real-world storage clusters.…”

Section: Related Workmentioning

confidence: 99%

PACEMAKER: Avoiding HeART attacks in storage clusters with disk-adaptive redundancy

Kadekodi,

Maturana,

Subramanya

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Data redundancy provides resilience in large-scale storage clusters, but imposes significant cost overhead. Substantial space-savings can be realized by tuning redundancy schemes to observed disk failure rates. However, prior design proposals for such tuning are unusable in real-world clusters, because the IO load of transitions between schemes overwhelms the storage infrastructure (termed transition overload).This paper analyzes traces for millions of disks from production systems at Google, NetApp, and Backblaze to expose and understand transition overload as a roadblock to diskadaptive redundancy: transition IO under existing approaches can consume 100% cluster IO continuously for several weeks. Building on the insights drawn, we present PACEMAKER, a low-overhead disk-adaptive redundancy orchestrator. PACE-MAKER mitigates transition overload by (1) proactively organizing data layouts to make future transitions efficient, and (2) initiating transitions proactively in a manner that avoids urgency while not compromising on space-savings. Evaluation of PACEMAKER with traces from four large (110K-450K disks) production clusters show that the transition IO requirement decreases to never needing more than 5% cluster IO bandwidth (0.2-0.4% on average). PACEMAKER achieves this while providing overall space-savings of 14-20% and never leaving data under-protected. We also describe and experiment with an integration of PACEMAKER into HDFS.1 AFR describes the expected fraction of disks that experience failure in a typical year.

show abstract

Section: Related Workmentioning

confidence: 99%

PACEMAKER: Avoiding HeART attacks in storage clusters with disk-adaptive redundancy

Kadekodi,

Maturana,

Subramanya

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Due to practical system constraints, changing n alone is typically insufficient and both n and k have to be changed simultaneously [9]. The resource cost of changing n and k on already encoded data can be prohibitively high and is a key barrier in the practical adoption of redundancy tuning [1]. Other reasons to change n and k on already encoded data might include variations in data popularity, failure rate uncertainty, or restrictions on the total amount of used storage.…”

Section: Introductionmentioning

confidence: 99%

“…The code conversion problem defined in [1] involves converting multiple stripes of an [n I , k I ] code (denoted by C I ) into (potentially multiple) stripes of an [n F , k F ] code (denoted by C F ), along with desired constraints on decodability such as both codes being Maximum Distance Separable (MDS). Considering multiple stripes enables code conversions to allow for changes in the code dimension (from k I to k F ).…”

Section: Introductionmentioning

confidence: 99%

“…Such tuning of redundancy requires code conversion, i.e., a change in code dimension and length on already encoded data. Convertible codes [1] are a new class of codes designed to perform such conversions efficiently. The access cost of conversion is the number of nodes accessed during conversion.Existing literature has characterized the access cost of conversion of linear MDS convertible codes only for a specific and small subset of parameters.…”

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Access-optimal Linear MDS Convertible Codes for All Parameters

Maturana

Mukka

Rashmi

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In large-scale distributed storage systems, erasure codes are used to achieve fault tolerance in the face of node failures. Tuning code parameters to observed failure rates has been shown to significantly reduce storage cost. Such tuning of redundancy requires code conversion, i.e., a change in code dimension and length on already encoded data. Convertible codes [1] are a new class of codes designed to perform such conversions efficiently. The access cost of conversion is the number of nodes accessed during conversion.Existing literature has characterized the access cost of conversion of linear MDS convertible codes only for a specific and small subset of parameters. In this paper, we present lower bounds on the access cost of conversion of linear MDS codes for all valid parameters. Furthermore, we show that these lower bounds are tight by presenting an explicit construction for access-optimal linear MDS convertible codes for all valid parameters. En route, we show that, one of the degrees-of-freedom in the design of convertible codes that was inconsequential in the previously studied parameter regimes, turns out to be crucial when going beyond these regimes and adds to the challenge in the analysis and code construction.This work was funded in part by an NSF CAREER award (CAREER-1943409) and a Google faculty research award. We thank Michael Rudow for his suggestions in the writing of this paper.F. Maturana and K. V. Rashmi are with the

show abstract

Locally Repairable Convertible Codes: Erasure Codes for Efficient Repair and Conversion

Maturana,

Rashmi

2023

2023 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

Convertible Codes: Enabling Efficient Conversion of Coded Data in Distributed Storage

Cited by 10 publications

References 80 publications

PACEMAKER: Avoiding HeART attacks in storage clusters with disk-adaptive redundancy

PACEMAKER: Avoiding HeART attacks in storage clusters with disk-adaptive redundancy

Access-optimal Linear MDS Convertible Codes for All Parameters

Locally Repairable Convertible Codes: Erasure Codes for Efficient Repair and Conversion

Contact Info

Product

Resources

About