General Sub-packetized Access-Optimal Regenerating Codes

Kralevska, Katina; Gligoroski, Danilo; Øverby, Harald

doi:10.1109/lcomm.2016.2561287

Cited by 19 publications

(33 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [183], the HDFS implementation of a class of MDS array codes called HashTag codes is discussed. The theoretical framework of HashTag codes was presented in [184]. These codes allow low sub-packetization levels at the expense of increased repair bandwidth and are designed to efficiently repair systematic nodes.…”

Section: Codes In Practicementioning

confidence: 99%

Erasure coding for distributed storage: an overview

Balaji

Krishnan

Vajha

et al. 2018

Sci. China Inf. Sci.

131

102

View full text Add to dashboard Cite

In a distributed storage system, code symbols are dispersed across space in nodes or storage units as opposed to time. In settings such as that of a large data center, an important consideration is the efficient repair of a failed node. Efficient repair calls for erasure codes that in the face of node failure, are efficient in terms of minimizing the amount of repair data transferred over the network, the amount of data accessed at a helper node as well as the number of helper nodes contacted. Coding theory has evolved to handle these challenges by introducing two new classes of erasure codes, namely regenerating codes and locally recoverable codes as well as by coming up with novel ways to repair the ubiquitous Reed-Solomon code. This survey provides an overview of the efforts in this direction that have taken place over the past decade. I. INTRODUCTIONThis survey article deals with the use of erasure coding for the reliable and efficient storage of large amounts of data in settings such as that of a data center. The amount of data stored in a single data center can run into tens or hundreds of petabytes. Reliability of data storage is ensured in part by introducing redundancy in some form, ranging from simple replication to the use of more sophisticated erasure-coding schemes such as Reed-Solomon codes. Minimizing the storage overhead that comes with ensuring reliability is a key consideration in the choice of erasure-coding scheme. More recently a second problem has surfaced, namely, that of node repair.In [1], [2] the authors study the Facebook warehouse cluster and analyze the frequency of node failures as well as the resultant network traffic relating to node repair. It was observed in [1] that a median of 50 nodes are unavailable per day and that a median of 180TB of cross-rack traffic is generated as a result of node unavailability. It was also reported that 98.08% of the cases have exactly one block missing in a stripe. The erasure code that was deployed in this instance was an [n = 14, k = 10] Reed Solomon (RS) code. Here n denotes the block length of the code and k the dimension. The conventional repair of an [n, k] RS code is inefficient in that the repair of a single node, calls for contacting k other (helper) nodes and downloading k times the amount of data stored in the failed node, which is clearly inefficient. Thus there is significant practical interest in the design of erasure-coding techniques that offer both low overhead and which can also be repaired efficiently.Coding theorists have responded to this need by coming up with two new classes of codes, namely ReGenerating (RG) and Locally Recoverable (LR) codes. The focus in a RG code is on minimizing the amount of data download needed to repair a failed node, termed the repair bandwidth while LR codes seek to minimize the number of helper nodes contacted for node repair, termed the repair degree. In a different direction, coding theorists have also re-examined the problem of node repair in RS codes and have come up with new and more efficient ...

show abstract

Section: Codes In Practicementioning

confidence: 99%

Erasure coding for distributed storage: an overview

Balaji

Krishnan

Vajha

et al. 2018

Sci. China Inf. Sci.

131

102

View full text Add to dashboard Cite

show abstract

“…α = r n/r , are introduced. A different approach to solve this problem is presented in [27], which is a method to reduce the subpacketization of the existing constructions dramatically in exchange for a slight increase in repair bandwidth.…”

Section: ) Msr Codementioning

confidence: 99%

Minimum Storage Regenerating Codes for All Parameters

Goparaju

Fazeli

Vardy

2017

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Abstract-Regenerating codes for distributed storage have attracted much research interest in the past decade. Such codes trade the bandwidth needed to repair a failed node with the overall amount of data stored in the network. Minimum storage regenerating (MSR) codes are an important class of optimal regenerating codes that minimize (first) the amount of data stored per node and (then) the repair bandwidth. Specifically, an [n, k, d]-(α) MSR code C over F q stores a file F consisting of αk symbols over F q among n nodes, each storing α symbols, in such a way that:• the file F can be recovered by downloading the content of any k of the n nodes; and • the content of any failed node can be reconstructed by accessing any d of the remaining n − 1 nodes and downloading α/(d−k+1) symbols from each of these nodes. In practice, the file F is typically available in uncoded form on some k of the n nodes, known as systematic nodes, and the defining node-repair condition above can be relaxed to requiring the optimal repair bandwidth for systematic nodes only. Such codes are called systematic-repair MSR codes.Unfortunately, finite-α constructions of [n, k, d] MSR codes are known only for certain special cases: either low rate, namely k/n 0.5, or high repair connectivity, namely d = n − 1. Our main result in this paper is a finite-α construction of systematic-repair [n, k, d] MSR codes for all possible values of parameters n, k, d. We also introduce a generalized construction for [n, k] MSR codes to achieve the optimal repair bandwidth for all values of d simultaneously.

show abstract

“…The scheduling algorithm for Def. 1 is presented in [4], [5]. Note that in the original presentation the indexing of the arrays is from 1 to r but in order to synchronize with the transformation of Li et al [9] here we use the indexing of the arrays from 0 to r − 1.…”

Section: A General Code Constructionmentioning

confidence: 99%

“…Table I summarizes several high-rate MDS codes with small sub-packetization [3]- [7]. Three piggyback designs were presented in [4]. For the purpose of this paper, we compare with piggyback design 2 that optimizes all-node repair for r ≥ 3 and sub-packetization of (2r − 3)m where m ≥ 1.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Explicit Construction of Systematic MDS Codes with Small Sub-Packetization for All-Node Repair

Kralevska

Gligoroski

2018

2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing,

Self Cite

View full text Add to dashboard Cite

An explicit construction of systematic MDS codes, called HashTag+ codes, with arbitrary sub-packetization level for all-node repair is proposed. It is shown that even for small subpacketization levels, HashTag+ codes achieve the optimal MSR point for repair of any parity node, while the repair bandwidth for a single systematic node depends on the sub-packetization level. Compared to other codes in the literature, HashTag+ codes provide from 20% to 40% savings in the average amount of data accessed and transferred during repair.

show abstract

General Sub-packetized Access-Optimal Regenerating Codes

Cited by 19 publications

References 10 publications

Erasure coding for distributed storage: an overview

Erasure coding for distributed storage: an overview

Minimum Storage Regenerating Codes for All Parameters

An Explicit Construction of Systematic MDS Codes with Small Sub-Packetization for All-Node Repair

Contact Info

Product

Resources

About