Data consistent up- and downstreaming in a distributed storage system

Sobe, Peter

doi:10.1145/1162618.1162621

Cited by 15 publications

(9 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An example is the jerasure [11] library that implements general matrix-based codes with optimizations for en-and decoding performance. A wide variety of XOR-based codes are implemented in the software-based distributed storage system NetRAID [14], [15]. With an equation-based description of encoding and decoding, this system allows a flexible definition of codes.…”

Section: Related Workmentioning

confidence: 99%

Pre-calculated equation-based decoding in failure-tolerant distributed storage

Sobe¹

2009

2009 IEEE International Symposium on Parallel &Amp; Distributed Processing

Self Cite

View full text Add to dashboard Cite

Data distribution together with erasure-tolerant codes allow to store data reliably, even with failed or temporarily disconnected storage resources. The encoding algorithm, i.e. the calculation of the codewords is expressed by XOR equations. Even decoding is the execution of a failurespecific set of equations that are build code-specifically and with knowledge of the failure situation. A new concept for a storage system is to provide encoding equations and decoding equations in advance, as a full description of the code which eliminates the calculations to obtain the recovery strategy. This concept includes that also decoding equations have to be provided in advance, for many different failure situations. This results in a large number of equations and may require a considerable amount of memory, but still a moderate amount -which can be traded for the gained flexibility and simplicity. In this paper we analyze the storage consumption of such a preprocessed decoding equation set. Furthermore, a data structure to access the required equations is proposed. It is shown that codes can be translated into equation sets that are used as parameter set by a storage system.

show abstract

Section: Related Workmentioning

confidence: 99%

Pre-calculated equation-based decoding in failure-tolerant distributed storage

Sobe¹

2009

2009 IEEE International Symposium on Parallel &Amp; Distributed Processing

Self Cite

View full text Add to dashboard Cite

show abstract

“…a secure network-transfer. This model is constructed with knowledge from the implementation of a distributed reliable storage system NetRAID [11,12]. Parameters are taken from experiments with this system to be practically relevant.…”

Section: Introductionmentioning

confidence: 99%

Combining Compression, Encryption and Fault-tolerant Coding for Distributed Storage

Sobe¹,

Peter

2007

2007 IEEE International Parallel and Distributed Processing Symposium

Self Cite

View full text Add to dashboard Cite

show abstract

“…A system that implements block-wise data distribution combined with configurable XOR and R/S redundancy is NetRAID [8,9]. Figure 1 depicts the commonly applied way for data storage.…”

Section: Introductionmentioning

confidence: 99%

Construction of efficient OR-based deletion - tolerant coding schemes

Sobe¹,

Peter²

2006

Proceedings 20th IEEE International Parallel &Amp; Distributed Processing Symposium

View full text Add to dashboard Cite

Fault-tolerant data layouts for storage systems are based on the principle to add redundancy to groups of data blocks and store them in different fault regions. Commonly, XOR-based codes are used with an optimal redundancy overhead but with the disadvantage of relatively high calculation costs. We present a scheme that encodes input data in a highly redundant code and exploits that redundancy for a fault tolerance scheme. It allows to recalculate missed bits in fewer steps than needed for XOR-based schemes. This simple and efficient en-and decoding requires an appropriate hardware architecture or a highly parallel microprocessor architecture. Particularly, disjunctions over many input bits must be calculated, e.g. by wide OR-gates or busses that are driven by multiple logic input lines. The high redundant encoding is combined with data compression for separated data streams, each stream dedicated to a storage device. The compression not only eliminates the introduced redundancy of the used code, it also eliminates redundancy in the input data.

show abstract

Data consistent up- and downstreaming in a distributed storage system

Cited by 15 publications

References 11 publications

Pre-calculated equation-based decoding in failure-tolerant distributed storage

Pre-calculated equation-based decoding in failure-tolerant distributed storage

Combining Compression, Encryption and Fault-tolerant Coding for Distributed Storage

Construction of efficient OR-based deletion - tolerant coding schemes

Contact Info

Product

Resources

About