Delayed Parity Generation in MDS Storage Codes

Mousavi, Sara; Zhou, Tiangang; Tian, Chao

doi:10.1109/isit.2018.8437700

Cited by 16 publications

(9 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given the generic nature of the proposed transformation, we anticipate it can be applied or extended to more cases and then lead to more desired storage codes. In fact, the code construction for delayed parity generation reported in [31] is indeed partly inspired by the generic transformation proposed here.…”

Section: Discussionmentioning

confidence: 97%

A Generic Transformation to Enable Optimal Repair in MDS Codes for Distributed Storage Systems

Tang

Tian

2018

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

For high-rate maximum distance separable (MDS) codes, most early constructions can only optimally repair all the systematic nodes but not for all the parity nodes initially. Fortunately, this issue was firstly solved by Li et al. in (IEEE Trans. Inform. Theory, 64(9), [6257][6258][6259][6260][6261][6262][6263][6264][6265][6266][6267] 2018), where a very powerful transformation that can convert any nonbinary MDS code into another MDS code with desired properties was proposed. However, the transformation does not work for binary MDS codes. In this note, we address this issue by proposing another generic transformation that can convert any (n, k) binary MDS code into a new binary MDS code, which endows any r = n − k chosen nodes with the optimal repair bandwidth and the optimal rebuilding access properties, and at the same time, preserves the normalized repair bandwidth and the normalized rebuilding access for the remaining k nodes under some conditions. As two immediate algorithms of this transformation, we show that 1) by applying the transformation multiple times, any (n, k) binary MDS code can be converted into an (n, k) binary MDS code with the optimal repair bandwidth and the optimal rebuilding access for all nodes, 2) any binary MDS code with the optimal repair bandwidth or the optimal rebuilding access for the systematic nodes only can be converted into an MDS code with the corresponding repair optimality for all nodes. Index TermsDistributed storage, high-rate, binary MDS codes, optimal rebuilding access, optimal repair bandwidth. the number of surviving nodes contacted during the repair process. Since 2010, various MDS codes with the optimal repair J. Li is with the ). 2 bandwidth have been proposed in the literature [8]-[27], where most works [9]-[22] consider the case d = n − 1 to maximally reduce the repair bandwidth, as γ * (d) is a decreasing function of d; this setting is also the focus of this work.However, at the practically more important range of high-rate case, i.e., k/n > 1/2, most early systematic code constructions can only optimally repair all the systematic nodes but not for all the parity nodes initially [9]- [17]. Fortunately, this issue was solved by Li et al.[28]-[30] firstly, where a very powerful transformation that can convert any nonbinary MDS code into another MDS code was proposed, which endows any r = n − k chosen nodes with the optimal repair bandwidth and the optimal rebuilding access properties, and at the same time, preserves the normalized repair bandwidth and the normalized rebuilding access for the remaining k nodes. The resultant code uses the same finite field as the base code. As two immediate applications of this transformation, it is showed that 1) any nonbinary MDS code with the optimal repair bandwidth or the optimal rebuilding access for the systematic nodes only can be converted into an MDS code with the corresponding repair optimality for all nodes, and 2) by applying the transformation multiple times, any (n, k) nonbinary scalar MDS code can be converted into a...

show abstract

Section: Discussionmentioning

confidence: 97%

A Generic Transformation to Enable Optimal Repair in MDS Codes for Distributed Storage Systems

Tang

Tian

2018

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

show abstract

“…While several works have considered the problem of designing erasure codes that allow transitions using less resources, existing solutions are limited to specific kinds of transitions and hence are not applicable in general. The case of adding parity chunks while keeping the number of data chunks fixed can be viewed [34,44,46] as the well-studied reconstruction problem, and hence the codes designed for optimal reconstruction (e.g., [10,18,38,45,46,61]) would lead to improved resource usage for this case. Several works have studied the case where the number of data nodes increases while the number of parity nodes remains fixed [23,41,65,67,70].…”

Section: Related Workmentioning

confidence: 99%

PACEMAKER: Avoiding HeART attacks in storage clusters with disk-adaptive redundancy

Kadekodi,

Maturana,

Subramanya

et al. 2021

Preprint

View full text Add to dashboard Cite

Data redundancy provides resilience in large-scale storage clusters, but imposes significant cost overhead. Substantial space-savings can be realized by tuning redundancy schemes to observed disk failure rates. However, prior design proposals for such tuning are unusable in real-world clusters, because the IO load of transitions between schemes overwhelms the storage infrastructure (termed transition overload).This paper analyzes traces for millions of disks from production systems at Google, NetApp, and Backblaze to expose and understand transition overload as a roadblock to diskadaptive redundancy: transition IO under existing approaches can consume 100% cluster IO continuously for several weeks. Building on the insights drawn, we present PACEMAKER, a low-overhead disk-adaptive redundancy orchestrator. PACE-MAKER mitigates transition overload by (1) proactively organizing data layouts to make future transitions efficient, and (2) initiating transitions proactively in a manner that avoids urgency while not compromising on space-savings. Evaluation of PACEMAKER with traces from four large (110K-450K disks) production clusters show that the transition IO requirement decreases to never needing more than 5% cluster IO bandwidth (0.2-0.4% on average). PACEMAKER achieves this while providing overall space-savings of 14-20% and never leaving data under-protected. We also describe and experiment with an integration of PACEMAKER into HDFS.1 AFR describes the expected fraction of disks that experience failure in a typical year.

show abstract

“…The closest related work [1] proposes the convertible codes framework considered in this work (discussed at length above). Several other works in the literature [10]- [14] have considered variants of the code conversion problem, largely within the context of so-called "regenerating codes" [15]. The study on regenerating codes, which are a class of codes that optimize for recovery for a small subset of nodes within a stripe (as opposed to decoding all original data), was initiated by Dimakis et al [15].…”

Section: Other Related Workmentioning

confidence: 99%

Access-optimal Linear MDS Convertible Codes for All Parameters

Maturana

Mukka

Rashmi

2020

Preprint

View full text Add to dashboard Cite

In large-scale distributed storage systems, erasure codes are used to achieve fault tolerance in the face of node failures. Tuning code parameters to observed failure rates has been shown to significantly reduce storage cost. Such tuning of redundancy requires code conversion, i.e., a change in code dimension and length on already encoded data. Convertible codes [1] are a new class of codes designed to perform such conversions efficiently. The access cost of conversion is the number of nodes accessed during conversion.Existing literature has characterized the access cost of conversion of linear MDS convertible codes only for a specific and small subset of parameters. In this paper, we present lower bounds on the access cost of conversion of linear MDS codes for all valid parameters. Furthermore, we show that these lower bounds are tight by presenting an explicit construction for access-optimal linear MDS convertible codes for all valid parameters. En route, we show that, one of the degrees-of-freedom in the design of convertible codes that was inconsequential in the previously studied parameter regimes, turns out to be crucial when going beyond these regimes and adds to the challenge in the analysis and code construction.This work was funded in part by an NSF CAREER award (CAREER-1943409) and a Google faculty research award. We thank Michael Rudow for his suggestions in the writing of this paper.F. Maturana and K. V. Rashmi are with the

show abstract

Delayed Parity Generation in MDS Storage Codes

Cited by 16 publications

References 15 publications

A Generic Transformation to Enable Optimal Repair in MDS Codes for Distributed Storage Systems

A Generic Transformation to Enable Optimal Repair in MDS Codes for Distributed Storage Systems

PACEMAKER: Avoiding HeART attacks in storage clusters with disk-adaptive redundancy

Access-optimal Linear MDS Convertible Codes for All Parameters

Contact Info

Product

Resources

About