A Robust Fault-Tolerant and Scalable Cluster-Wide Deduplication for Shared-Nothing Storage Systems

Khan, Awais; Lee, Chang-Gyu; Hamandawana, Prince; Park, Sung-Yong; Kim, Young-Jae

doi:10.1109/mascots.2018.00016

Cited by 16 publications

(15 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The decision to select the deduplication mode lies within the administrators' choice and the needs of their systems. In this paper, since inline deduplication provides instant storage space savings, we adopted a recent inline cluster-wide inline deduplication design proposed in [7].…”

Section: Deduplication Storage Clustermentioning

confidence: 99%

“…Since cluster-wide deduplication has been adopted in the general storage community [7], [8], [9], [10], [11], [12], [13], [14], it is necessary to extend its advantages to the ML and DL storage architectures without affecting performance. Deduplication improves the storage capacity by removing duplicate data across the cluster.…”

Section: Introductionmentioning

confidence: 99%

“…We design and implement the proposed approach using a Ceph back-end storage with inline deduplication enabled [7]. We perform experiments on different ML platforms with different neural networks (AlexNet and LSTM-64).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Accelerating ML/DL Applications with Hierarchical Caching on Deduplication Storage Clusters

Hamandawana

Khan

Kim

et al. 2021

IEEE Trans. Big Data

Self Cite

View full text Add to dashboard Cite

Large scale machine learning (ML) and deep learning (DL) platforms face challenges when integrated with deduplication enabled storage clusters. In the quest to achieve smart and efficient storage utilization, removal of duplicate data introduces bottlenecks, since deduplication alters the I/O transaction layout of the storage system. Therefore, it is critical to address such deduplication overhead for acceleration of ML/DL computation in deduplication storage. Existing state of the art ML/DL storage solutions such as Alluxio and AutoCache adopt non deduplication-aware caching mechanisms, which lacks the much needed performance boost when adopted in deduplication enabled ML/DL clusters. In this paper, we introduce Redup, which eliminates the performance drop caused by enabling deduplication in ML/DL storage clusters. At the core, is a Redup Caching Manager (RDCM), composed of a 2-tier deduplication layoutaware caching mechanism. The RDCM provides an abstraction of the underlying deduplication storage layout to ML/DL applications and provisions a decoupled acceleration of object reconstruction during ML/DL read operations. Our Redup evaluation shows negligible performance drop in ML/DL training performances as compared to a cluster without deduplication, whilst significantly outperforming Alluxio and AutoCache in terms of various performance metrics.

show abstract

Section: Deduplication Storage Clustermentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Accelerating ML/DL Applications with Hierarchical Caching on Deduplication Storage Clusters

Hamandawana

Khan

Kim

et al. 2021

IEEE Trans. Big Data

Self Cite

View full text Add to dashboard Cite

show abstract

“…Khan et al [ 93 ] presented a robust, fault-tolerant, and scalable cluster-wide deduplication that was able to eliminate duplicate copies across the entire cluster. The evaluation showed great savings in disk space with minimal performance degradation, as well as great robustness in the event of sudden server failure.…”

Section: Description Of the Set Of Work By Categorymentioning

confidence: 99%

Application of Dynamic Fragmentation Methods in Multimedia Databases: A Review

Castro-Medina

Rodríguez-Mazahua

López-Chau

et al. 2020

Entropy

View full text Add to dashboard Cite

Fragmentation is a design technique widely used in multimedia databases, because it produces substantial benefits in reducing response times, causing lower execution costs in each operation performed. Multimedia databases include data whose main characteristic is their large size, therefore, database administrators face a challenge of great importance, since they must contemplate the different qualities of non-trivial data. These databases over time undergo changes in their access patterns. Different fragmentation techniques presented in related studies show adequate workflows, however, some do not contemplate changes in access patterns. This paper aims to provide an in-depth review of the literature related to dynamic fragmentation of multimedia databases, to identify the main challenges, technologies employed, types of fragmentation used, and characteristics of the cost model. This review provides valuable information for database administrators by showing essential characteristics to perform proper fragmentation and to improve the performance of fragmentation schemes. The reduction of costs in fragmentation methods is one of the most desired main properties. To fulfill this objective, the works include cost models, covering different qualities. In this analysis, a set of characteristics used in the cost models of each work is presented to facilitate the creation of a new cost model including the most used qualities. In addition, different data sets or reference points used in the testing stage of each work analyzed are presented.

show abstract

“…To meet HDFS application equipment, deduplication schemes are proposed as necessary. Khan et al [31] proposed a multilevel pattern-matching algorithm for deduplication of big data. MLPMA is implemented based on the similarity and locality of data, and the Bloom filter is applied to the deduplication cluster to achieve effective data deletion.…”

Section: Hdfs Hash Functionmentioning

confidence: 99%

Data Deduplication Technology for Cloud Storage

Bian

Shao

et al. 2020

Teh. vjesn.

View full text Add to dashboard Cite

With the explosive growth of information data, the data storage system has stepped into the cloud storage era. Although the core of the cloud storage system is distributed file system in solving the problem of mass data storage, a large number of duplicate data exist in all storage system. File systems are designed to control how files are stored and retrieved. Fewer studies focus on the cloud file system deduplication technologies at the application level, especially for the Hadoop distributed file system. In this paper, we design a file deduplication framework on Hadoop distributed file system for cloud application developer. Proposed RFD-HDFS and FD-HDFS two data deduplication solutions process data deduplication online, which improves storage space utilisation and reduces the redundancy. In the end of the paper, we test the disk utilisation and the file upload performance on RFD-HDFS and FD-HDFS, and compare HDFS with the disk utilisation of two system frameworks. The results show that the two-system framework not only implements data deduplication function but also effectively reduces the disk utilisation of duplicate files. So, the proposed framework can indeed reduce the storage space by eliminating redundant HDFS file.

show abstract

A Robust Fault-Tolerant and Scalable Cluster-Wide Deduplication for Shared-Nothing Storage Systems

Cited by 16 publications

References 10 publications

Accelerating ML/DL Applications with Hierarchical Caching on Deduplication Storage Clusters

Accelerating ML/DL Applications with Hierarchical Caching on Deduplication Storage Clusters

Application of Dynamic Fragmentation Methods in Multimedia Databases: A Review

Data Deduplication Technology for Cloud Storage

Contact Info

Product

Resources

About