A Logistic Based Mathematical Model to Optimize Duplicate Elimination Ratio in Content Defined Chunking Based Big Data Storage System

Wang, Longxiang; Dong, Xiaoshe; Zhang, Xingjun; Guo, Fuliang; Wang, Yinfeng; Gong, Wei

doi:10.3390/sym8070069

Cited by 7 publications

(2 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The previously suggested 4 KB or 8 KB chunk size did not offer the best optimization for the deduplication ratio, so Wang et al 38 presented a mathematical model based on logistics to improve the deduplication ratio based on the expected chunk size. They used two real‐world datasets to verify that the model was accurate, and the results showed that the R2 value was higher than 0.9.…”

Section: Related Workmentioning

confidence: 99%

Blockchain‐based data deduplication using novel content‐defined chunking algorithm in cloud environment

Prakash J,

et al. 2023

Int J Network Mgmt

View full text Add to dashboard Cite

The cloud environment is inherently dynamic as users are added immensely in a short duration. It is indeed difficult to manage such user profiles and associated data. Meanwhile, the cloud data expand at a twofold‐to‐threefold rate on average, making storage space management and data integrity maintenance a mandatory task but also risky. The main approaches for addressing these data maintenance challenges in a cloud context are deduplication and data protection. In order to manage storage space, finding and removing identical copies of the same data from the cloud are possible, resulting in a reduction in the amount of storage space needed. Furthermore, duplicate copies are considerably reduced in cloud storage owing to data deduplication. Here, a decentralized ledger public blockchain network is introduced to protect the Integrity of data stored in cloud storage. This research proposes data deduplication using speedy content‐defined Chunking (SpeedyCDC) algorithm in the public blockchain. Many people and businesses outsource sensitive data to remote cloud servers because it considerably eliminates the hassle of managing software and infrastructure. However, the ownership and control rights of users data are nonetheless divided because it is outsourced to cloud storage providers (CSPs) and kept on a distant cloud. As a result, users have a great deal of difficulty in verifying the Integrity of sensitive data. Analysis using datasets from Geospatial Information Systems (GIS) revealed that the throughput increased by 5%–6% over that of the fastCDC technique, which offered Integrity since a blockchain network secured it.

show abstract

Section: Related Workmentioning

confidence: 99%

Blockchain‐based data deduplication using novel content‐defined chunking algorithm in cloud environment

Prakash J,

et al. 2023

Int J Network Mgmt

View full text Add to dashboard Cite

show abstract

“…During the implementation of CDC, the vital parameter, called expected chunk size, will definitely and significantly affect the duplicate elimination ratio (DER). For an improvement, Wang et al [21] uncovered the hidden relationship between DER and expected chunk size through designing a logistic based mathematical model to provide a theoretical basis for this kind of method. Their experimental results showed that the logistic mathematical way is correct and reasonable for choosing the expected chunk size and reaching the goal of optimizing DER.…”

Section: Related Workmentioning

confidence: 99%

Self-Adaptive Pre-Processing Methodology for Big Data Stream Mining in Internet of Things Environmental Sensor Monitoring

et al. 2017

View full text Add to dashboard Cite

Over the years, advanced IT technologies have facilitated the emergence of new ways of generating and gathering data rapidly, continuously, and largely and are associated with a new research and application branch, namely, data stream mining (DSM). Among those multiple scenarios of DSM, the Internet of Things (IoT) plays a significant role, with a typical meaning of a tough and challenging computational case of big data. In this paper, we describe a self-adaptive approach to the pre-processing step of data stream classification. The proposed algorithm allows different divisions with both variable numbers and lengths of sub-windows under a whole sliding window on an input stream, and clustering-based particle swarm optimization (CPSO) is adopted as the main metaheuristic search method to guarantee that its stream segmentations are effective and adaptive to itself. In order to create a more abundant search space, statistical feature extraction (SFX) is applied after variable partitions of the entire sliding window. We validate and test the effort of our algorithm with other temporal methods according to several IoT environmental sensor monitoring datasets. The experiments yield encouraging outcomes, supporting the reality that picking significant appropriate variant sub-window segmentations heuristically with an incorporated clustering technique merit would allow these to perform better than others.

show abstract

Predicting Deduplication Performance: An Analytical Model and Empirical Evaluation

Randall

2022

2022 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

A Logistic Based Mathematical Model to Optimize Duplicate Elimination Ratio in Content Defined Chunking Based Big Data Storage System

Cited by 7 publications

References 8 publications

Blockchain‐based data deduplication using novel content‐defined chunking algorithm in cloud environment

Blockchain‐based data deduplication using novel content‐defined chunking algorithm in cloud environment

Self-Adaptive Pre-Processing Methodology for Big Data Stream Mining in Internet of Things Environmental Sensor Monitoring

Predicting Deduplication Performance: An Analytical Model and Empirical Evaluation

Contact Info

Product

Resources

About