Asymptotically Optimal Regenerating Codes Over Any Field

Raviv, Netanel

doi:10.1109/tit.2018.2815537

Cited by 2 publications

(2 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…and studies the existence of MBR codes with inherent double replication, for all parameters. In [17], the authors provide regenerating-code constructions that asymptotically achieve the MSR or MBR point as k increases and these codes can be constructed over any field, provided the file size is large enough. In [18], the authors introduce some extensions to the classical MBR framework by permitting the presence of a certain number of error-prone nodes during repair/reconstruction and by introducing flexibility in choosing the parameter d during node repair.…”

Section: Other Workmentioning

confidence: 99%

Erasure coding for distributed storage: an overview

Balaji

Krishnan

Vajha

et al. 2018

Sci. China Inf. Sci.

131

102

View full text Add to dashboard Cite

In a distributed storage system, code symbols are dispersed across space in nodes or storage units as opposed to time. In settings such as that of a large data center, an important consideration is the efficient repair of a failed node. Efficient repair calls for erasure codes that in the face of node failure, are efficient in terms of minimizing the amount of repair data transferred over the network, the amount of data accessed at a helper node as well as the number of helper nodes contacted. Coding theory has evolved to handle these challenges by introducing two new classes of erasure codes, namely regenerating codes and locally recoverable codes as well as by coming up with novel ways to repair the ubiquitous Reed-Solomon code. This survey provides an overview of the efforts in this direction that have taken place over the past decade. I. INTRODUCTIONThis survey article deals with the use of erasure coding for the reliable and efficient storage of large amounts of data in settings such as that of a data center. The amount of data stored in a single data center can run into tens or hundreds of petabytes. Reliability of data storage is ensured in part by introducing redundancy in some form, ranging from simple replication to the use of more sophisticated erasure-coding schemes such as Reed-Solomon codes. Minimizing the storage overhead that comes with ensuring reliability is a key consideration in the choice of erasure-coding scheme. More recently a second problem has surfaced, namely, that of node repair.In [1], [2] the authors study the Facebook warehouse cluster and analyze the frequency of node failures as well as the resultant network traffic relating to node repair. It was observed in [1] that a median of 50 nodes are unavailable per day and that a median of 180TB of cross-rack traffic is generated as a result of node unavailability. It was also reported that 98.08% of the cases have exactly one block missing in a stripe. The erasure code that was deployed in this instance was an [n = 14, k = 10] Reed Solomon (RS) code. Here n denotes the block length of the code and k the dimension. The conventional repair of an [n, k] RS code is inefficient in that the repair of a single node, calls for contacting k other (helper) nodes and downloading k times the amount of data stored in the failed node, which is clearly inefficient. Thus there is significant practical interest in the design of erasure-coding techniques that offer both low overhead and which can also be repaired efficiently.Coding theorists have responded to this need by coming up with two new classes of codes, namely ReGenerating (RG) and Locally Recoverable (LR) codes. The focus in a RG code is on minimizing the amount of data download needed to repair a failed node, termed the repair bandwidth while LR codes seek to minimize the number of helper nodes contacted for node repair, termed the repair degree. In a different direction, coding theorists have also re-examined the problem of node repair in RS codes and have come up with new and more efficient ...

show abstract

Section: Other Workmentioning

confidence: 99%

Erasure coding for distributed storage: an overview

Balaji

Krishnan

Vajha

et al. 2018

Sci. China Inf. Sci.

131

102

View full text Add to dashboard Cite

show abstract

“…Hence, the presented coding scheme in this work is providing the bandwidth adaptivity property at no extra cost in the field size requirements, and any field of size larger than n could be used as the code alphabet. It worth mentioning that the field size requirement of the only other bandwidth adaptive exact repair MSR constructions, introduced in [3] is lower bounded by n 2 − kn which is significantly larger for large distributed storage networks n. Moreover, techniques such as those presented in [37] could easily be applied in the presented coding scheme to reduce the field size to any arbitrary (e.g. binary) small field.…”

Section: A Field Size Requirementmentioning

confidence: 99%

Product Matrix MSR Codes With Bandwidth Adaptive Exact Repair

Mahdaviani

Mohajer

Khisti

2018

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

In a distributed storage systems (DSS) with k systematic nodes, robustness against node failure is commonly provided by storing redundancy in a number of other nodes and performing repair mechanism to reproduce the content of the failed nodes. Efficiency is then achieved by minimizing the storage overhead and the amount of data transmission required for data reconstruction and repair, provided by coding solutions such as regenerating codes [1]. Common explicit regenerating code constructions enable efficient repair through accessing a predefined number, d, of arbitrary chosen available nodes, namely helpers. In practice, however, the state of the system dynamically changes based on the request load, the link traffic, etc., and the parameters which optimize system's performance vary accordingly. It is then desirable to have coding schemes which are able to operate optimally under a range of different parameters simultaneously. Specifically, adaptivity in the number of helper nodes for repair is of interest. While robustness requires capability of performing repair with small number of helpers, it is desirable to use as many helpers as available to reduce the transmission delay and total repair traffic.In this work we focus on the minimum storage regenerating (MSR) codes, where each of the n nodes in the network is supposed to store α information units, and the source data of size kα could be recovered from any arbitrary set of k nodes. We introduce a class of MSR codes that realize optimal repair bandwidth simultaneously with a set of different choices for the number of helpers, namely D = {d1, · · · , d δ }. Our coding scheme follows the Product Matrix (PM) framework introduced in [2], and could be considered as a generalization of the PM MSR code presented in [2], such that any di = (i + 1)(k − 1) helpers can perform an optimal repair. As a result, the coding rate in our construction is limited by k n ≤ 1 2 . However, similar to the original design of PM MSR codes, our solution can realize practical values of the parameter α. Recently [3] has presented another explicit MSR coding scheme which is capable of performing optimal repair for various number of helpers. The solution presented in [3] works for any arbitrary set of parameters k, D and can achieve high coding rates, but the required α for this code is exponentially large. We show that the required value for α in the coding scheme presented in this work is exponentially smaller when compared to [3] for the same set of other parameters. Particularly, for a DSS with n nodes, and k systematic nodes, the required value for α is reduced from s n to sk, where s = lcm (d1 − k + 1, · · · , d δ − k + 1). We also show the required field size in the presented coding scheme is equal to n 1 .

show abstract

Asymptotically Optimal Regenerating Codes Over Any Field

Cited by 2 publications

References 33 publications

Erasure coding for distributed storage: an overview

Erasure coding for distributed storage: an overview

Product Matrix MSR Codes With Bandwidth Adaptive Exact Repair

Contact Info

Product

Resources

About