Improving Data Availability in HDFS through Replica Balancing

Fazul, Rhauani Weber Aita; Cardoso, Paulo Vinicius; Barcelos, Patricia Pitthan

doi:10.1109/ladc48089.2019.8995674

Cited by 9 publications

(10 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…• initial RPP, in which the replicas are placed immediately after writing the files based on the standard Replica Placement Policy of the HDFS (i.e., without reactive balancing), as presented in Section 2.1; • datanode, in which the redistribution of the blocks is done by the HDFS Balancer configured with the default balancing policy (Section 4); • blockpool, in which the rearrangement of the replicas is performed by HDFS Balancer configured with the blockpool policy (Section 4); and • custom, in which the replica balancing process in the file system is conducted by the HDFS Balancer customized with the "data availability" priority presented in [Fazul et al 2019], as pointed out in Section 5.…”

Section: Experiments and Discussionmentioning

confidence: 99%

“…In previous work [Fazul et al 2019], we proposed a customized balancing policy for the HDFS Balancer, which focuses on improving data availability and performance through replica balancing. To this end, a balancing priority, called "data availability", has been incorporated into the block move scheduling step executed in each balancing iteration to prioritize block movements that increase the availability of the data stored in the HDFS, that is, place the replicas in as many racks as possible.…”

Section: Related Workmentioning

confidence: 99%

“…There are two policies implemented by the HDFS Balancer, namely, datanode and blockpool [Cloudera, Inc. 2021]. Furthermore, customized policies designed to improve the balancing process and incorporate new features into the HDFS Balancer can be found in the literature [Fazul et al 2019].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A practical analysis of balancing policies for rearranging data replicas in HDFS clusters

Fazul

Barcelos²

2022

Anais Do XXIII Simpósio Em Sistemas Computacionais De Alto Desempenho (SSCAD 2022)

Self Cite

View full text Add to dashboard Cite

Data replication is the main fault tolerance mechanism implemented by the HDFS. The placement of the replicated data across the nodes directly influences replica balancing and data locality, which are essential to ensure high reliability and data availability. The HDFS Balancer is the official solution to perform replica balancing through data redistribution. In this work, we conducted a practical experiment to evaluate different policies for replica rearrangement, namely: datanode, blockpool, and custom. The evaluation results underline the behavior and the effectiveness of each policy. In addition, we investigated the cost of the HDFS Balancer operation and the performance and availability improvements promoted by a balanced replica distribution.

show abstract

Section: Experiments and Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A practical analysis of balancing policies for rearranging data replicas in HDFS clusters

Fazul

Barcelos²

2022

Anais Do XXIII Simpósio Em Sistemas Computacionais De Alto Desempenho (SSCAD 2022)

Self Cite

View full text Add to dashboard Cite

show abstract

“…IV. CUSTOMIZED REPLICA BALANCING POLICY Previous work attested that replicas imbalance directly affect the HDFS performance in serving I/O bound applications [12]. Motivated by the achieved results, we defined a customized policy for the HDFS Balancer dedicated to optimizing the default operation policy of the Hadoop's native balancer.…”

Section: A Hdfs Balancermentioning

confidence: 99%

The Node Status as a Prioritization Strategy for Replica Balancing in a HDFS Cluster

Fazul

Barcelos

2020

Anais Estendidos Do X Simpósio Brasileiro De Engenharia De Sistemas Computacionais (SBESC Estendido 2020)

View full text Add to dashboard Cite

Data replication is the main fault tolerance mechanism of HDFS, the Hadoop Distributed File System. Although replication is essential to ensure high availability and reliability, the replicas might not always be placed evenly among the nodes. The HDFS Balancer is an integrated solution of Apache Hadoop that performs replica balancing through the rearrangement of the data blocks stored in the file system. The Balancer, however, demands a high computational effort of the nodes during its operation. This work presents a customization for the HDFS Balancer that considers the status of the nodes as a strategy to minimize the overhead caused by the balancing operation in the cluster. To this end, metrics obtained at runtime are used as a way to prioritize the nodes during data redistribution, making it occurs primarily between nodes with low communication traffic. Also, the Balancer starts to operate aiming at a minimum balance level, reducing the number of data transfers required to even up the data stored in the cluster. The evaluation results showed that the proposed customization allows reducing the time and bandwidth needed to reach the system balance.

show abstract

“…Este trabalho destina-seà investigação do comportamento da política customizada considerando o uso simultâneo de duas prioridades: "confiabilidade dos racks" e "disponibilidade dos dados". Em trabalhos anteriores, as prioridades de confiabilidade e disponibilidade foram avaliadas de forma totalmente independente [Fazul et al 2019a, Fazul et al 2019b. A associação destas prioridades, por sua vez,é algo inédito na política customizada e abre um conjunto de novas otimizações para o balanceamento de réplicas, apresentando-se assim como a contribuição principal do presente artigo.…”

Section: Distribuição Dos Dados Disponibilidade Dos Dadosunclassified

A Confiabilidade dos Racks e a Disponibilidade dos Dados como Métricas para o Balanceamento de Réplicas de um Cluster HDFS

Fazul¹,

Barcelos²

2020

Anais XXXVIII Simpósio Brasileiro De Redes De Computadores E Sistemas Distribuídos (SBRC 2020)

Self Cite

View full text Add to dashboard Cite

A replicação, embora vital para o bom funcionamento do HDFS, favorece o desbalanceamento do cluster. O HDFS Balancer é uma solução integrada ao Apache Hadoop voltada ao balanceamento de réplicas. O balanceador, entretanto, não é otimizado para atender possíveis demandas de tolerância a falhas e disponibilidade dos dados ao realizar a redistribuição das réplicas. Este trabalho apresenta uma customização para o HDFS Balancer que considera a propensão a falhas dos racks do sistema para definir quais nodos devem receber uma maior ou menor quantidade de dados. Em conjunto, utilizou-se uma prioridade que esforça-se em aumentar a disponibilidade geral dos dados armazenados no HDFS durante o reposicionamento das réplicas no cluster.

show abstract

Improving Data Availability in HDFS through Replica Balancing

Cited by 9 publications

References 3 publications

A practical analysis of balancing policies for rearranging data replicas in HDFS clusters

A practical analysis of balancing policies for rearranging data replicas in HDFS clusters

The Node Status as a Prioritization Strategy for Replica Balancing in a HDFS Cluster

A Confiabilidade dos Racks e a Disponibilidade dos Dados como Métricas para o Balanceamento de Réplicas de um Cluster HDFS

Contact Info

Product

Resources

About