A data integrity verification service for cloud storage based on building blocks

Reyes-Anastacio, Hugo G.; Gonzalez-Compeán, J. L.; Morales-Sandoval, Miguel; Carretero, Jesús

doi:10.1109/csit.2018.8486274

Cited by 7 publications

(7 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The primary objective of big data transfer frameworks is to ensure the accuracy of transferred data. To accomplish this, extensive research has been conducted on the design and optimization of data integrity verification systems in different contexts, including storage systems [29,30], cloud-based storage [31,32], file systems [33,34], and data transfer systems [35][36][37][38].…”

Section: Data Integritymentioning

confidence: 99%

CRBF: Cross-Referencing Bloom-Filter-Based Data Integrity Verification Framework for Object-Based Big Data Transfer Systems

2023

View full text Add to dashboard Cite

Various components are involved in the end-to-end path of data transfer. Protecting data integrity from failures in these intermediate components is a key feature of big data transfer tools. Although most of these components provide some degree of data integrity, they are either too expensive or inefficient in recovering corrupted data. This problem highlights the need for application-level end-to-end integrity verification during data transfer. However, the computational, memory, and storage overhead of big data transfer tools can be a significant bottleneck for ensuring data integrity due to the large size of the data. This paper proposes a novel framework for data integrity verification in big data transfer systems using a cross-referencing Bloom filter. This framework has three advantages over state-of-the-art data integrity techniques: lower computation and memory overhead and zero false-positive errors for a limited number of elements. This study evaluates the computation, memory, recovery time, and false-positive overhead for the proposed framework and compares them with state-of-the-art solutions. The evaluation results indicate that the proposed framework is efficient in detecting and recovering from integrity errors while eliminating false positives in the Bloom filter data structure. In addition, we observe negligible computation, memory, and recovery overheads for all workloads.

show abstract

Section: Data Integritymentioning

confidence: 99%

CRBF: Cross-Referencing Bloom-Filter-Based Data Integrity Verification Framework for Object-Based Big Data Transfer Systems

2023

View full text Add to dashboard Cite

show abstract

“…Ensuring the correctness of the transferred data is one of the primary concerns of data transfer frameworks. Many studies have been performed on the design and implementation of data integrity verification and their optimization in data storage [34,36,37], cloud storage [38][39][40][41][42], file systems [32][33][34][35]43], databases [32,44,45], and data transfer systems [18,19,46,47].…”

Section: Related Workmentioning

confidence: 99%

TPBF: Two-Phase Bloom-Filter-Based End-to-End Data Integrity Verification Framework for Object-Based Big Data Transfer Systems

2022

View full text Add to dashboard Cite

Computational science simulations produce huge volumes of data for scientific research organizations. Often, this data is shared by data centers distributed geographically for storage and analysis. Data corruption in the end-to-end route of data transmission is one of the major challenges in distributing the data geographically. End-to-end integrity verification is therefore critical for transmitting such data across data centers effectively. Although several data integrity techniques currently exist, most have a significant negative influence on the data transmission rate as well as the storage overhead. Therefore, existing data integrity techniques are not viable solutions in high performance computing environments where it is very common to transfer huge volumes of data across data centers. In this study, we propose a two-phase Bloom-filter-based end-to-end data integrity verification framework for object-based big data transfer systems. The proposed solution effectively handles data integrity errors by reducing the memory and storage overhead and minimizing the impact on the overall data transmission rate. We investigated the memory, storage, and data transfer rate overheads of the proposed data integrity verification framework on the overall data transfer performance. The experimental findings showed that the suggested framework had 5% and 10% overhead on the total data transmission rate and on the total memory usage, respectively. However, we observed significant savings in terms of storage requirements, when compared with state-of-the-art solutions.

show abstract

“…This scheme was deployed in a prototype where the components such as the deduplication system, the IDA component, and the hash algorithm h (SHA-3-256 [18]) were implemented mainly in C language. The clients implementing the data preparation and retrieval schemes were also developed in C. The integrity verification metadata service was developed as a web service and uses a Mongo database deployed in the cloud [23]. The building blocks and the parallel patterns were created by using a framework based on Kulla [24].…”

Section: Prototype Implementation Detailsmentioning

confidence: 99%

A Federated Content Distribution System to Build Health Data Synchronization Services

Carrizales-Espinoza

Sánchez‐Gallegos

Gonzalez-Compeán

et al. 2021

2021 29th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP)

Self Cite

View full text Add to dashboard Cite

In organizational environments, such as in hospitals, data have to be processed, preserved, and shared with other organizations in a cost-efficient manner. Moreover, organizations have to accomplish different mandatory non-functional requirements imposed by the laws, protocols, and norms of each country. In this context, this paper presents a Federated Content Distribution System to build infrastructure-agnostic health data synchronization services. In this federation, each hospital manages local and federated services based on a pub/sub model. The local services manage users and contents (i.e., medical imagery) inside the hospital, whereas federated services allow the cooperation of different hospitals sharing resources and data. Data preparation schemes were implemented to add non-functional requirements to data. Moreover, data published in the content distribution system are automatically synchronized to all users subscribed to the catalog where the content was published.

show abstract

A data integrity verification service for cloud storage based on building blocks

Cited by 7 publications

References 10 publications

CRBF: Cross-Referencing Bloom-Filter-Based Data Integrity Verification Framework for Object-Based Big Data Transfer Systems

CRBF: Cross-Referencing Bloom-Filter-Based Data Integrity Verification Framework for Object-Based Big Data Transfer Systems

TPBF: Two-Phase Bloom-Filter-Based End-to-End Data Integrity Verification Framework for Object-Based Big Data Transfer Systems

A Federated Content Distribution System to Build Health Data Synchronization Services

Contact Info

Product

Resources

About