Cooperative pipelined regeneration in distributed storage systems

Abstract

In distributed storage systems, a substantial volume of data are stored in a distributed fashion, across a large number of storage nodes. To maintain data integrity, when existing storage nodes fail, lost data are regenerated at replacement nodes. Regenerating multiple data losses in batches can reduce the consumption of bandwidth. However, existing schemes are only able to achieve lower bandwidth consumption by utilizing a large number of participating nodes. In this paper, we propose a cooperative pipelined regeneration process that regenerates multiple data losses cooperatively with much fewer participating nodes. We show that cooperative pipelined regeneration is not only able to maintain optimal data integrity, but also able to further reduce the consumption of bandwidth as well.

HeMatch: A redundancy layout placement scheme for erasure-coded storages in practical heterogeneous failure patterns

“…Khan et al propose Rotated Reed Solomon Codes for RAID in cloud file systems, which consider the read patterns and make the parities split across adjacent rows to reduce the unnecessary penalty during most reconstruction cases [15]. Li et al propose a cooperative pipelined regeneration process that accesses multiple unavailable data cooperatively [16]. A single pipelined regeneration process is also proposed for a single failure for Minimum-Storage Regenerating Codes [17].…”

Erasure coding for cloud storage systems: A survey

“…To achieve this goal, Li et al [35,36] proposed a pipelined repair with minimumstorage regenerating codes. The principle of pipelined repair is that providers of consecutive newcomers are selected to be overlapped, because any k or more than k different providers are sufficient in the repair.…”

“…The .n; k; d; t / cooperative pipelined regenerating codes, which can be regarded as a generalization of independent pipelined regenerating codes, have been discussed in Ref. [36], which require p t.d C t / t providers while maintaining the bandwidth consumption of .n; k; d; t / cooperative regenerating codes.…”

“…When providers are overlapped, only three providers are required in each round of repair while one newcomer can be fully repaired after each round of repair. However, it can be shown that this overhead is marginal in practical cloud storage systems [36] . Moreover, every block in fully repaired newcomers can still maintain the MDS property, even though the number of providers in the repair can be less than k.…”

“…In return, the number of providers can be significantly saved. Apart from regenerating codes, Li et al [36] found that random linear codes could also be applied in the pipelined repair. In this section, we discuss the erasure codes that are more explicitly optimized towards this property.…”