Storage-Optimized Data-Atomic Algorithms for Handling Erasures and Errors in Distributed Storage Systems

Konwar, Kishori M.; Prakash, Naveen; Kantor, Erez; Lynch, Nancy; Médard, Muriel; Schwarzmann, Alexander A.

doi:10.1109/ipdps.2016.55

Cited by 11 publications

(21 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Erasure coding based algorithms for read/write memory emulation with atomic (linearizable) consistency are developed for both crash faults and byzantine faults in [13,25,11,24,17,21,6]. These algorithms partition a data object and encode the data partitions -that is, they do not use cross-object erasure coding.…”

Section: Summary Of Contributionsmentioning

confidence: 99%

“…So when one object, say X 1 is updated to a newer value, say x 1 , CausalEC uses certain properties of the encoding functions to transform the codeword symbol 6 to x 1 + x 2 + x 3 . Another crucial conceptual difference from previous works [11,24,21,13,25,27,6] is that they typically involve multiple rounds of communication between clients and servers allowing a client to reconcile the differences in the versions stored at different servers. In fact, typically, algorithms either require a larger number of rounds of communication (e.g., [13,6,21]) or a higher amount of data (e.g., [24,21,11,25,1]) to reconcile this difference.…”

Section: Summary Of Contributionsmentioning

confidence: 99%

“…Another crucial conceptual difference from previous works [11,24,21,13,25,27,6] is that they typically involve multiple rounds of communication between clients and servers allowing a client to reconcile the differences in the versions stored at different servers. In fact, typically, algorithms either require a larger number of rounds of communication (e.g., [13,6,21]) or a higher amount of data (e.g., [24,21,11,25,1]) to reconcile this difference. In our algorithm design, the communication requirements are much more stringent.…”

Section: Summary Of Contributionsmentioning

confidence: 99%

“…Erasure Coding Based Consistent Data Stores: Erasure coding based algorithms for read/write memory emulation with atomic (linearizable) consistency are developed for both crash faults and byzantine faults in [13,25,11,24,17,21]. There are also erasure coding based systems [14,34,44,20] that adapt consensus algorithms to utilize erasure coding and provide more involved data access primitives to the clients.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

CausalEC: A Causally Consistent Data Storage Algorithm based on Cross-Object Erasure Coding

Cadambe¹,

Lyu²

2021

Preprint

View full text Add to dashboard Cite

Causally consistent distributed storage systems have received significant recent attention due to the potential for providing a low latency data access as compared with linearizability. Current causally consistent data stores use partial or full replication to ensure data access to clients over a distributed setting. In this paper, we develop, for the first time, an erasure coding based algorithm called CausalEC that ensures causal consistency for a collection of read-write objects stored in a distributed set of nodes over an asynchronous message passing system. CausalEC can use an arbitrary linear erasure code for data storage, and ensures liveness and storage properties prescribed by the erasure code.CausalEC retains a key benefit of previously designed replication-based algorithms -every write operation is "local", that is, a server performs only local actions before returning to a client that issued a write operation. For servers that store certain objects in an uncoded manner, read operations to those objects also return locally. In general, a read operation to an object can be returned by a server on contacting a small subset of other servers so long as the underlying erasure code allows for the object to be decoded from that subset. As a byproduct, we develop EventualEC, a new eventually consistent erasure coding based data storage algorithm.A novel technical aspect of CausalEC is that it uses cross-object erasure coding, where nodes encode values across multiple objects, unlike previous consistent erasure coding based solutions. CausalEC navigates the technical challenges of cross-object erasure coding, in particular, pertaining to re-encoding the objects when writes update the values and ensuring that reads are served in the transient state where the system transitions to storing the codewords corresponding to the new object versions.

show abstract

Section: Summary Of Contributionsmentioning

confidence: 99%

Section: Summary Of Contributionsmentioning

confidence: 99%

Section: Summary Of Contributionsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

CausalEC: A Causally Consistent Data Storage Algorithm based on Cross-Object Erasure Coding

Cadambe¹,

Lyu²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The broadcast primitive was briefly discussed in Section III. It we use the implementation in [17], a broadcast message is received within a duration of 2τ 0 . Thus, a write completes within a duration of 4τ 1 + 2τ 0 .…”

Section: Proof Of Theorem Iv9 [Atomicity]mentioning

confidence: 99%

A Layered Architecture for Erasure-Coded Consistent Distributed Storage

Konwar

Prakash

Lynch

et al. 2017

Proceedings of the ACM Symposium on Principles of Distributed Computing

Self Cite

View full text Add to dashboard Cite

Motivated by emerging applications to the edge computing paradigm, we introduce a two-layer erasure-coded fault-tolerant distributed storage system offering atomic access for read and write operations. In edge computing, clients interact with an edge-layer of servers that is geographically near; the edge-layer in turn interacts with a back-end layer of servers. The edge-layer provides low latency access and temporary storage for client operations, and uses the back-end layer for persistent storage. Our algorithm, termed Layered Data Storage (LDS) algorithm, offers several features suitable for edge-computing systems, works under asynchronous message-passing environments, supports multiple readers and writers, and can tolerate f 1 < n 1 /2 and f 2 < n 2 /3 crash failures in the two layers having n 1 and n 2 servers, respectively. We use a class of erasure codes known as regenerating codes for storage of data in the back-end layer. The choice of regenerating codes, instead of popular choices like Reed-Solomon codes, not only optimizes the cost of back-end storage, but also helps in optimizing communication cost of read operations, when the value needs to be recreated all the way from the back-end. The two-layer architecture permits a modular implementation of atomicity and erasure-code protocols; the implementation of erasure-codes is mostly limited to interaction between the two layers. We prove liveness and atomicity of LDS, and also compute performance costs associated with read and write operations. In a system with n 1 = Θ(n 2 ), f 1 = Θ(n 1 ), f 2 = Θ(n 2 ), the write and read costs are respectively given by Θ(n 1 ) and Θ(1) + n 1 I(δ > 0). Here δ is a parameter closely related to the number of write operations that are concurrent with the read operation, and I(δ > 0) is 1 if δ > 0, and 0 if δ = 0. The cost of persistent storage in the back-end layer is Θ(1). The impact of temporary storage is minimally felt in a multi-object system running N independent instances of LDS, where only a small fraction of the objects undergo concurrent accesses at any point during the execution. For the multi-object system, we identify a condition on the rate of concurrent writes in the system such that the overall storage cost is dominated by that of persistent storage in the back-end layer, and is given by Θ(N ).

show abstract

Fragmented Objects: Boosting Concurrency of Shared Large Objects

Anta

Georgiou

Hadjistasi

et al. 2021

Structural Information and Communication Complexity

View full text Add to dashboard Cite

This work examines strategies to handle large shared data objects in distributed storage systems (DSS), while boosting the number of concurrent accesses, maintaining strong consistency guarantees, and ensuring good operation performance. To this respect, we define the notion of fragmented objects: concurrent objects composed of a list of fragments (or blocks) that allow operations to manipulate each of their fragments individually. As the fragments belong to the same object, it is not enough that each fragment is linearizable to have useful consistency guarantees in the composed object. Hence, we capture the consistency semantic of the whole object with the notion of fragmented linearizability. Then, considering that a variance of linearizability, coverability, is more suited for versioned objects like files, we provide an implementation of a distributed file system, called COBFS, that utilizes coverable fragmented objects (i.e., files). In COBFS, each file is a linked-list of coverable block objects. Preliminary emulation of COBFS demonstrates the potential of our approach in boosting the concurrency of strongly consistent large objects.

show abstract

Storage-Optimized Data-Atomic Algorithms for Handling Erasures and Errors in Distributed Storage Systems

Cited by 11 publications

References 20 publications

CausalEC: A Causally Consistent Data Storage Algorithm based on Cross-Object Erasure Coding

CausalEC: A Causally Consistent Data Storage Algorithm based on Cross-Object Erasure Coding

A Layered Architecture for Erasure-Coded Consistent Distributed Storage

Fragmented Objects: Boosting Concurrency of Shared Large Objects

Contact Info

Product

Resources

About