Efficient updates in cross-object erasure-coded storage systems

Esmaili, Kyumars Sheykh; Chiniah, Aatish; Datta, Anwitaman

doi:10.1109/bigdata.2013.6691658

Cited by 14 publications

(13 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluate the update efficiency by the Update Time, which is defined as the total duration from the update request to the end of the update. We consider three standard configurations of ( , ) code in HDFS-RAID: (1) RS (9,6) [ =9, =6, =3] (2) RS (10,6) [ =10, =6, =4] (3) RS (14,10) [ =14, =10, =4] and test the update efficiency of the four schemes under different cross-rack bandwidths. In actual production, the intra-rack bandwidth is 10Gb/s, the cross-rack bandwidth is 1Gb/s [25].…”

Section: ) Update Efficiency Assessmentmentioning

confidence: 99%

SDCUP: Software-Defined-Control Based Erasure-Coded Collaborative Data Update Mechanism

Chen

Zhong

2020

IEEE Access

View full text Add to dashboard Cite

The storage system often uses erasure coding to provide the necessary fault tolerance. The erasure-coded update involves data transmission and data calculation of multiple nodes. Frequent updates will cause massive communication overhead. This paper mainly considers two issues: (1) In the scenario of frequent small-size updates, there are repetitive behaviors upon update, which causes bandwidth consumption to increase exponentially as the number of update nodes increases; (2) with the increase of data scale, there are local link busy phenomena caused by unbalanced use of links during update, which can prone to bottleneck links. In order to improve the inefficient update due to network bottlenecks. We propose SDCUP, a software-defined-control collaborative update mechanism that reduce update time for erasure-coded data with network load balance. Specially, SDCUP uses the software-defined control method to select the update transmission path according to the actual link load and adjust the data flow transmission rate by monitoring the degree of network load balance periodically. To further reduce the cross-rack update traffic, SDCUP unloads the calculation to the switch to realize the data aggregation in the rack, and it parallelizes sub-update operations to efficiently and cooperatively update. To evaluate the performance of SDCUP, we conduct simulation experiments on Mininet with real-world traces. The simulation results show that SDCUP can achieve a better load balance in multiple scenarios. Compared with the other data update schemes, the proposed method can improve the system throughput by up to 21% and reduce the update time by up to 47%. INDEX TERMS Erasure coding, software-defined, data update, network load balance.

show abstract

Section: ) Update Efficiency Assessmentmentioning

confidence: 99%

SDCUP: Software-Defined-Control Based Erasure-Coded Collaborative Data Update Mechanism

Chen

Zhong

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…For the two-level DEC, the total number of I/O reads and the storage size are random variables that are respectively given by η = k + L j=2 η j , where η j is given in (11) and δ = n + L j=2 n j , where n j is given in (10). Note that η and δ are also dependent on the threshold T .…”

Section: Threshold Design Problemmentioning

confidence: 99%

“…In this paper, we investigate a new aspect of erasure code design, aimed at storing multiple versions of data. The presented work is loosely related to the issues of efficient updates [7], [8], [9], [10], and of deduplication [11]. Existing works on update of • J. Harshan and Frédérique Oggier are with the Division of Mathematical Sciences, Nanyang Technological University, Singapore.…”

Section: Introductionmentioning

confidence: 99%

Differential Erasure Codes for Efficient Archival of Versioned Data in Cloud Storage Systems

Harshan

Datta

Oggier

2016

Transactions on Large-Scale Data- And Knowledge-Centered Systems XXX

Self Cite

View full text Add to dashboard Cite

In this paper, we study the problem of storing an archive of versioned data in a reliable and efficient manner in distributed storage systems. We propose a new storage technique called differential erasure coding (DEC) where the differences (deltas) between subsequent versions are stored rather than the whole objects, akin to a typical delta encoding technique. However, unlike delta encoding techniques, DEC opportunistically exploits the sparsity (i.e., when the differences between two successive versions have few non-zero entries) in the updates to store the deltas using compressed sensing techniques applied with erasure coding. We first show that DEC provides significant savings in the storage size for versioned data whenever the update patterns are characterized by in-place alterations. Subsequently, we propose a practical DEC framework so as to reap storage size benefits against not just in-place alterations but also real-world update patterns such as insertions and deletions that alter the overall data sizes. We conduct experiments with several synthetic workloads to demonstrate that the practical variant of DEC provides significant reductions in storage overhead (up to 60% depending on the workload) compared to baseline storage system which incorporates concepts from Rsync, a delta encoding technique to store and synchronize data across a network.

show abstract

“…Let {x j ∈ F k q , 1 ≤ j ≤ L} be the sequence of versions of a data object to be stored in the network, where x j is the jth version (or version at the j-th instant of time). The number of components modified from x j to x j+1 is reflected in the vector z j+1 = x j+1 − x j in (2) which is then γ j+1 -sparse (see Definition 1) for some 1 ≤ γ j+1 ≤ k. We propose an encoding strategy using an (n, k) linear erasure code (see (1)) which exploits the sparsity of the differences across updates, thus referred to as sparsity exploiting coding (SEC). Note that the value γ j+1 may a priori vary across updates of the same object and across different objects, and that sparsity is exploitable only when γ j+1 < k 2 .…”

Section: Sparsity Exploiting Coding (Sec)mentioning

confidence: 99%

“…Existing works are however predominantly geared towards storing immutable content, unlike the case of versioned data. The recent works which do focus on mutable content do so in the context of efficiently carrying out an update [7], [3], [5], [1], and thus focuses only on the storage of the latest version of the data.…”

Section: Introductionmentioning

confidence: 99%

Sparsity Exploiting Erasure Coding for Resilient Storage and Efficient I/O Access in Delta Based Versioning Systems

Harshan

Oggier

Datta

2015

2015 IEEE 35th International Conference on Distributed Computing Systems

Self Cite

View full text Add to dashboard Cite

Abstract-In this paper we study the problem of storing reliably an archive of versioned data. Specifically, we focus on systems where the differences (deltas) between subsequent versions rather than the whole objects are stored -a typical model for storing versioned data. For reliability, we propose erasure encoding techniques that exploit the sparsity of information in the deltas while storing them reliably in a distributed back-end storage system, resulting in improved I/O read performance to retrieve the whole versioned archive. Along with the basic techniques, we propose a few optimization heuristics, and evaluate the techniques' efficacy analytically and with numerical simulations.

show abstract

Efficient updates in cross-object erasure-coded storage systems

Cited by 14 publications

References 9 publications

SDCUP: Software-Defined-Control Based Erasure-Coded Collaborative Data Update Mechanism

SDCUP: Software-Defined-Control Based Erasure-Coded Collaborative Data Update Mechanism

Differential Erasure Codes for Efficient Archival of Versioned Data in Cloud Storage Systems

Sparsity Exploiting Erasure Coding for Resilient Storage and Efficient I/O Access in Delta Based Versioning Systems

Contact Info

Product

Resources

About