Available Storage Space Sensitive Replica Placement Strategy of HDFS

Abstract

When HDFS runs for a period of time, the default HDFS replica placement strategy does not consider heterogeneous nodes and randomly selectes datanodes, so available storage space of each datanode are differenced from several G to tens of G range, HDFS may have some available storage space smaller datanode. If HDFS continues to write replica in these available storage space smaller datanodes,it may cause the failure of replica placement; These nodes run MapReduce tasks, MapReduce task may fail due to smaller storage space available. Therefore, this paper proposes an Available storage space sensitive replica placement strategy of HDFS,this strategy uses available storage space and current number of connections of per datanode to compute evaluation value, then chooses the biggest evaluation value datanode as the best datanode to place replica. The experimental results show that the strategy achieves placing replica in accordance with available storage space of datanode, effectively avoiding the available storage space smaller datanode, reducing the probability of writing replica failed because the available storage space is too small, shorten the file upload time.

Overview of Storage Architecture and Strategy of HDFS

“…HDFS namenode automatically selects datanode to save a copy of the data. In the actual business, there are the following scenarios [7][8][9][10]:…”