A Dynamic Metadata Equipotent Subtree Partition Policy for Mass Storage System

Abstract

Metadata plays an important role in mass storage system. How to distribute and balance the metadata of the metadata server cluster determines the overall performance of a cluster. Two popular metadata distribution policies are the dynamic subtree policy and hashing policy, while the dynamic subtree partition is vulnerable to the imbalance workload and hashing partition has a random distribution which will incur a burst of network overhead when updating metadata.We present a novel approach for metadata management. It combines hash and subtree partitioning policies together to partition directory hierarchy tree into equipotent subtrees with a certain granularity and employs value of hashing subtree to distribute subtrees across the metadata servers. It also employs a balance strategy to adjust the metadata distribution dynamically. After adjustment, we present a hot spots elimination strategy to detect and reclaim hot spots in the file system efficiently. We also demonstrate a design using this strategy to achieve more efficient performance than the other policies using hashing partitioning and subtree partitioning purely.