Pipelined Regeneration with Regenerating Codes for Distributed Storage Systems

Abstract

Distributed storage systems store a substantial amount of data and maintain data integrity by storing redundancy in a large number of storage nodes. When storage nodes fail, the lost data should be regenerated on replacement nodes. Regenerating codes minimize the volume of network traffic brought by the regeneration process. However, regenerating codes consume less traffic by requiring a larger number of nodes participating in the regeneration process. In this paper, we propose a new pipelined regeneration process for regenerating codes, that enables the system to consume a smaller amount of network traffic during the regeneration without engaging a large number of participating nodes. Moreover, the time spent during the regeneration can be further saved, while the data integrity can still be maintained no worse than the conventional regeneration process with regenerating codes.

HeMatch: A redundancy layout placement scheme for erasure-coded storages in practical heterogeneous failure patterns

“…Li et al propose a cooperative pipelined regeneration process that accesses multiple unavailable data cooperatively [16]. A single pipelined regeneration process is also proposed for a single failure for Minimum-Storage Regenerating Codes [17]. To tolerate simultaneous device failures and sector failures, a series of good works have been proposed, including the IDR scheme by Dholakia et al [18], SD codes by Plank et al [19], and the STAIR codes by Li et al [20].…”

Erasure coding for cloud storage systems: A survey

“…To achieve this goal, Li et al [35,36] proposed a pipelined repair with minimumstorage regenerating codes. The principle of pipelined repair is that providers of consecutive newcomers are selected to be overlapped, because any k or more than k different providers are sufficient in the repair.…”

“…6, the number of providers can be reduced by 67%. In fact, the independent .n; k; d / regenerating codes for the pipelined repair [35] can reduce the number of participating nodes to as few as 2 p d C 1 1 in which only p d C 1 1 are providers, while still maintaining the same bandwidth consumption of .n; k; d / minimum-storage regenerating codes. It is surprising to find that the number of providers can be even less than k, as long as the newcomer is able to receive data from at least d providers before its "graduation".…”

Cooperative pipelined regeneration in distributed storage systems

“…A pipelined regeneration process with MSR codes [22] has been proposed for a single failure. However, it fails to utilize multiple newcomers cooperatively, and can only support MSR codes.…”