A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems

Plank, James S.

doi:10.1002/(sici)1097-024x(199709)27:9<995::aid-spe111>3.0.co;2-6

Cited by 261 publications

(209 citation statements)

References 21 publications

Supporting

Mentioning

204

Contrasting

Unclassified

Order By: Relevance

“…Still another possible approach is to use erasure coding, such as Rabin's Information Dispersal Algorithm (IDA) [48,3] and Reed Soloman Coding [34,45], to disperse a file into a set of pieces such that any sufficient subset allows reconstruction. Consequently, this approach is usually more space-efficient and reliable than RAID-5 and mirroring.…”

Section: The Choice Of Fault Tolerance Designsmentioning

confidence: 99%

CEFT: A cost-effective, fault-tolerant parallel virtual file system

Zhu

Jiang

2006

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

The vulnerability of computer nodes due to component failures is a critical issue for cluster-based file systems. This paper studies the development and deployment of mirroring in cluster-based parallel virtual file systems to provide fault tolerance and analyzes the tradeoffs between the performance and the reliability in the mirroring scheme. It presents the design and implementation of CEFT, a scalable RAID-10 style file system based on PVFS, and proposes four novel mirroring protocols depending on whether the mirroring operations are server-driven or client-driven, whether they are asynchronous or synchronous. The comparisons of their write performances, measured in a real cluster, and their reliability and availability, obtained through analytical modeling, show that these protocols strike different tradeoffs between the reliability and performance. Protocols with higher peak write performance are less reliable than those with lower peak write performance, and vice versa. A hybrid protocol is proposed to optimize this tradeoff.

show abstract

Section: The Choice Of Fault Tolerance Designsmentioning

confidence: 99%

CEFT: A cost-effective, fault-tolerant parallel virtual file system

Zhu

Jiang

2006

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

show abstract

“…[9]). Therefore, Reed-Solomon (RS) codes have become very popular in distributed storage systems [15,11] and disk arrays [6,14] since they combine a good rate of (n − k)/k with distance d(Y ) = n − k + 1. Unfortunately, as RS codes are MDS codes, they also suffer from an undesired update overhead because if x is modified, all blocks of y must be rewritten, what is dismal in a SAN suffering from expensive I/O accesses.…”

Section: Related Workmentioning

confidence: 99%

“…This property has made erasure codes become very prominent in many application areas [1,7,16]. In storage networks such as RAID-arrays [13,14] and modern storage area networks (SAN) [10] access to hard disks is comparably slow, and thus, data is scattered into fixed sized blocks which are evenly distributed among the storage devices to exploit access parallelism. If then some disks fail for reading (erasures), in the optimal case, any k symbols from y are sufficient to recover x, i.e.…”

Section: Introductionmentioning

confidence: 99%

“…such codes can tolerate up to n − k erasures which may be caused by failed, respectively temporarily not accessible disks. Since the number n of blocks (symbols) in a codeword y is fixed (called data stripe) and all blocks in a stripe are hosted by n different disk, linear block codes are mainly applied [13,5,14,8]. More importantly, linear block codes are optimal codes, i.e.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Read-Write-Codes: An Erasure Resilient Encoding System for Flexible Reading and Writing in Storage Networks

Mense

Schindelhauer

2009

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. We introduce the Read-Write-Coding-System (RWC) -a very flexible class of linear block codes that generate efficient and flexible erasure codes for storage networks. In particular, given a message x of k symbols and a codeword y of n symbols, an RW code defines additional parameters k ≤ r, w ≤ n that offer enhanced possibilities to adjust the fault-tolerance capability of the code. More precisely, an RWC provides linear (n, k, d)-codes that have (a) minimum distance d = n−r+1 for any two codewords, and (b) for each codeword there exists a codeword for each other message with distance of at most w. Furthermore, depending on the values r, w and the code alphabet, different block codes such as parity codes (e.g. RAID 4/5) or Reed-Solomon (RS) codes (if r = k and thus, w = n) can be generated. In storage networks in which I/O accesses are very costly and redundancy is crucial, this flexibility has considerable advantages as r and w can optimally be adapted to read or write intensive applications; only w symbols must be updated if the message x changes completely, what is different from other codes which always need to rewrite y completely as x changes. In this paper, we first state a tight lower bound and basic conditions for all RW codes. Furthermore, we introduce special RW codes in which all mentioned parameters are adjustable even online, that is, those RW codes are adaptive to changing demands. At last, we point out some useful properties regarding safety and security of the stored data.

show abstract

“…Vandermonde matrices are commonly used in creating systematic Reed-Solomon codes for RAID schemes recovering from disk failures (see [10]). For this, one could start with an (N m + K) × K Vandermonde matrix and then apply elementary matrix operations to bring it into a systematic form.…”

Section: System Architecture and Encoding/decoding Schemementioning

confidence: 99%

Scalable Ubiquitous Data Access in Clustered Sensor Networks

Lee

Thomo

et al.

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Wireless sensor networks have drawn much attention due to their ability to monitor ecosystems and wildlife habitats. In such systems, the data should be intelligently collected to avoid human intervention. For this, we propose a network infrastructure in which the sensor nodes are designated as "data-generating" or "data-storage" nodes. Data-generating nodes take measurements, whereas data-storage nodes make themselves available to compute and store checksums of data received from nearby data-generating nodes. We propose a spatially-clustered architecture for our storage nodes and a coding scheme that allows a data collector to recover all original data by querying only a small random subset of storage nodes from each cluster. The size of such a subset is equal to the number of data-generating nodes that the cluster serves. When the clustering structure of the storage nodes is unknown, we show that recovering of the original data is still possible if a random subset of the right size of storage nodes is selected for querying. We determine this right size so as to have a successful decoding with a probability exceeding a given threshold.

show abstract

A tutorial on Reed-Solomon coding for fault-tolerance in RAID-like systems

Cited by 261 publications

References 21 publications

CEFT: A cost-effective, fault-tolerant parallel virtual file system

CEFT: A cost-effective, fault-tolerant parallel virtual file system

Read-Write-Codes: An Erasure Resilient Encoding System for Flexible Reading and Writing in Storage Networks

Scalable Ubiquitous Data Access in Clustered Sensor Networks

Contact Info

Product

Resources

About