Research on Multifeature Data Routing Strategy in Deduplication

Abstract

Deduplication is a popular data reduction technology in storage systems which has significant advantages, such as finding and eliminating duplicate data, reducing data storage capacity required, increasing resource utilization, and saving storage costs. The file features are a key factor that is used to calculate the similarity between files, but the similarity calculated by the single feature has some limitations especially for the similar files. The storage node feature reflects the load condition of the node, which is the key factor to be considered in the data routing. This paper introduces a multifeature data routing strategy (DRMF). The routing strategy is made based on the features of the cluster, including routing communication, file similarity calculation, and the determination of the target node. The mutual information exchange is achieved by routing communication, routing servers, and storage nodes. The storage node calculates the similarity between the files stored, and then the file is routed according to the information provided by the routing server. The routing server determines the target node of the route according to the similar results and the node load features. The system prototype is designed and implemented; also, we develop a system to process the feature of cluster and determine the specific parameters of various features of experiments. In the end, we simulate the multifeature data routing and single-feature data routing, respectively, and compare the deduplication rate and data slope between the two strategies. The experimental results show that the proposed data routing strategy using multiple features can improve the deduplication rate of the cluster and maintain a lower data skew rate compared with the single-feature-based routing strategy MCS; DRMF can improve the deduplication rate of the cluster and maintain a lower data skew rate.

Research on Routing Strategy in Cluster Deduplication System

“…Figure 4-1 shows the deduplication rate between the stateful EMC-stateful, stateless EMC-stateless, and DRMF [19] algorithms based on the EMC scheme and the data frequency-based classification routing algorithm DRDF.…”

“…This scheme can obtain a high deduplication rate under the premise of maintaining a balanced data distribution . However, similar to the broadcast system communication overhead and frequent block fingerprint query, it seriously affects the deduplication performance of the cluster [18][19].…”

“…Figure 4-2 shows the logic time comparison of the DRDF and DRMF [19] and EMC stateful and stateless routing algorithms for processing about 11 million fingerprint data. EM C-stateful has to query storage nodes every time routing data, so it needs more time overhead.…”

Research on Data Routing Strategy of Deduplication in Cloud Environment

“…Firstly, we test the comparison of the deduplication rate of the algorithms proposed in this paper under different storage nodes. As shown in Figure 4-1, Stateful denotes the stateful data routing policy, DRMF [19] algorithms and Σ-Dedupe [12] algorithms, Stateless denotes the stateless data routing policy proposed in this paper, and DBF denotes the global BloomFilter-based data routing policy proposed in this paper. In the experiments, five storage nodes, 8 nodes, 16 nodes, 32 nodes, 64 nodes, 128 nodes, 256 nodes, and 512 nodes, are selected to verify the number of nodes.…”

“…Both approaches have their pros and cons, but to ensure the parallelism of the system and low system overhead, the current cluster deduplication system mainly uses the former for data routing [3][4][5]. Implement the routing strategy requires maintaining fingerprint indexes of data blocks in the memory of storage nodes, which the size of data block is 4KB and the fingerprint size of each data block is about 40B.…”