Towards Server-side Repair for Erasure Coding-based Distributed Storage Systems

Abstract

Erasure coding is one of the main mechanisms to add redundancy in a distributed storage system, by which a file with k data segments is encoded into a file with n coded segments such that any k coded segments can be used to recover the original k data segments. Each coded segment is stored at a storage server. Under an adversarial setting in which the storage servers can exhibit Byzantine behavior, remote data checking (RDC) can be used to ensure that the stored data remains retrievable over time. The main previous RDC scheme to offer such strong security guarantees, HAIL, has an inefficient repair procedure, which puts a high load on the data owner when repairing even one corrupt data segment.In this work, we propose RDC-EC, a novel RDC scheme for erasure code-based distributed storage systems that can function under an adversarial setting. With RDC-EC we offer a solution to an open problem posed in previous work and build the first such system that has an efficient repair phase. The main insight is that RDC-EC is able to reduce the load on the data owner during the repair phase (i.e., lower bandwidth and computation) by shifting most of the burden from the data owner to the storage servers during repair. RDC-EC is able to maintain the advantages of systematic erasure coding: optimal storage for a certain reliability level and sub-file access. We build a prototype for RDC-EC and show experimentally that RDC-EC can handle efficiently large amounts of data.

Sanitizing data is not enough!

“…The solution could be to introduce redundancy. In storage domain, multiple coding techniques like replication [20], erasure coding [16] and network coding [17] can be utilized to add redundancy. Considering both the MAP table and the key table are small in size as well as their dynamic nature, we can simply use replication and create duplicate copies for both of them.…”

Efficient Publicly Verifiable Proofs of Data Replication and Retrievability Applicable for Cloud Storage

“…By using erasure codes, data retrievability and reliability are guaranteed among distributed storage servers, and clients can detect and repair file corruption. The authors in [41] achieve a more efficient mechanism for repairs by extending HAIL [33] and moving bulk computations to the cloud provider. The authors in [42] present a scheme for remote data checking in network coding-based distributed storage systems.…”

What If Keys Are Leaked? towards Practical and Secure Re-Encryption in Deduplication-Based Cloud Storage

“…Deduplication is a widely used technique in the cloud environment [4], aiming at removing cross-user duplicates. Note that the deduplication does not contradict with the known durability technique, which replicates data redundantly [20][21][22][23] such that data can be always recoverable upon being corrupted. This is because the redundant data created for durability purposes usually belong to a single user, and the deduplication technique removes redundant data cross users, which were unknown by users and have not been used for durability purposes.…”