Rack Aware Data Placement for Network Consumption in Erasure-Coded Clustered Storage Systems

Abstract

Abstract:The amount of encoded data replication in an erasure-coded clustered storage system has a great impact on the bandwidth consumption and network latency, mostly during data reconstruction. Aimed at the reasons that lead to the excess data transmission between racks, a rack aware data block placement method is proposed. In order to ensure rack-level fault tolerance and reduce the frequency and amount of the cross-rack data transmission during data reconstruction, the method deploys partial data block concentration to store the data blocks of a file in fewer racks. Theoretical analysis and simulation results show that our proposed strategy greatly reduces the frequency and data volume of the cross-rack transmission during data reconstruction. At the same time, it has better performance than the typical random distribution method in terms of network usage and data reconstruction efficiency.

Storage and repair bandwidth tradeoff for distributed storage systems with clusters and separate nodes

“…In [1], [23], the authors also investigated cluster DSSs with relay nodes which were not only used for node repair, but also used for data collection. In [25], the authors proposed a data placement method to reduce the cross-cluster bandwidth on data reconstruction in the cluster DSS model without relay nodes. The DSS model with two clusters was considered in [6], [22].…”