Distributed data provenance for large-scale data-intensive computing

Zhao, Dongfang; Shou, Chen; Malik, Tanu; Raicu, Ioan

doi:10.1109/cluster.2013.6702685

Cited by 41 publications

(20 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By decoupling metadata and data, we are able to apply flexible strategies on metadata management and data I/Os. Prior work [23,24] also shows that a distributed hash table offers a flexible yet efficient means for tracking applications' provenance.…”

Section: Discussionmentioning

confidence: 99%

Virtual chunks: On supporting random accesses to scientific data in compressible storage systems

Zhao

Yin

Qiao

et al. 2014

2014 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

Abstract-Data compression could ameliorate the I/O pressure of scientific applications on high-performance computing systems. Unfortunately, the conventional wisdom of naively applying data compression to the file or block brings the dilemma between efficient random accesses and high compression ratios. Filelevel compression can barely support efficient random accesses to the compressed data: any retrieval request need trigger the decompression from the beginning of the compressed file. Block-level compression provides flexible random accesses to the compressed data, but introduces extra overhead when applying the compressor to each every block that results in a degraded overall compression ratio. This paper introduces a concept called virtual chunks aiming to support efficient random accesses to the compressed scientific data without sacrificing its compression ratio. In essence, virtual chunks are logical blocks identified by appended references without breaking the physical continuity of the file content. These additional references allow the decompression to start from an arbitrary position (efficient random access), and retain the file's physical entirety to achieve high compression ratio on par with file-level compression. One potential concern of virtual chunks lies on its space overhead (from the additional references) that degrades the compression ratio, but our analytic study and experimental results demonstrate that such overhead is negligible. We have implemented virtual chunks in two forms: a middleware to the GPFS parallel file system, and a module in the FusionFS distributed file system. Large-scale evaluations on up to 1,024 cores showed that virtual chunks could help improve the I/O throughput by 2X speedup.

show abstract

Section: Discussionmentioning

confidence: 99%

Virtual chunks: On supporting random accesses to scientific data in compressible storage systems

Zhao

Yin

Qiao

et al. 2014

2014 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

show abstract

“…More recently, Zhao et al [48] proposed using both a distributed hash table (FusionFS [49]) and a centralized database (SPADE [47]) to manage the metadata. Similarly to us, their metadata model includes both file operations and provenance information.…”

Section: Related Workmentioning

confidence: 99%

Efficient Scheduling of Scientific Workflows Using Hot Metadata in a Multisite Cloud

Liu

Villaseñor-Pineda

Pacitti

et al. 2019

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Ordonnancement Efficace de Workflows Scientifiques en exploitant les Mé tadonné es Chaudes dans un Cloud Multisite Ré suméLes applications scientifiques à grande é chelle sont souvent exprimé es sous forme de workflows scientifiques (SWfs) qui aident à dé finir les jobs de traitement des donné es et les dé pendances entre les activité s des jobs. Certains SWfs né cessitent une trè s grande quantité de stockage et de calcul, ce qui peut ê tre obtenu en exploitant plusieurs data centers dans un cloud. Dans ce contexte, la gestion des mé tadonné es et l'ordonnancement des tâches entre différents data centers deviennent critiques pour l'exécution efficace de SWf. Dans cet article, nous proposons une architecture et un modè le distribué s hybrides, en utilisant les mé tadonné es chaudes (fré quemment consultées) pour l'ordonnancement efficace de SWf dans un cloud multisite. Nous utilisons notre modè le dans un systè me de gestion de workflows scientifiques (SWfMS) pour valider et ré gler son applicabilité à diffé rents workflows scientifiques ré els avec diffé rents algorithmes d'ordonnancement. Nous montrons que la combinaison d'une gestion efficace des métadonnées chaudes et des algorithmes d'ordonnancement améliore les performances du SWfMS. En é vitant les opé rations inutiles de mé tadonné es froides, le temps d'exé cution des jobs qui s'exécutent en parallèle est ré duit jusqu'à 64,1% et celui de l'ensemble des workflows scientifiques jusqu'à 37,5%. ABSTRACTLarge-scale scientific applications are often expressed as scientific workflows (SWfs) that help defining data processing jobs and dependencies between jobs' activities. Several SWfs have huge storage and computation requirements, and so they need to be processed in multiple (cloud-federated) datacenters. It has been shown that efficient metadata handling plays a key role in the performance of computing systems. However, most of this evidence concern only single-site, HPC systems to date. In addition, the efficient scheduling of tasks among different data centers is critical to the SWf execution. In this paper, we present a hybrid distributed model and architecture, using hot metadata (frequently accessed metadata) for efficient SWf scheduling in a multisite cloud. We couple our model with a scientific workflow management system (SWfMS) to validate and tune its applicability to different real-life scientific workflows with different scheduling algorithms. We show that the combination of efficient management of hot metadata and scheduling algorithms improves the performance of SWfMS, reducing the execution time of highly parallel jobs up to 64.1% and that of the whole scientific workflows up to 37.5%, by avoiding unnecessary cold metadata operations. KEYWORDShot metadata, metadata management, multisite clouds, scientific workflows, geo-distributed applications.

show abstract

“…We believe HyCache+, together with other features such as data compression [52] and data provenance [37,49], would make the next generation extreme-scale storage system (e.g. [47]) more practical for real applications [30].…”

Section: F Broader Impactmentioning

confidence: 99%

HyCache+: Towards Scalable High-Performance Caching Middleware for Parallel File Systems

Zhao

Qiao

Raicu

2014

2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Self Cite

View full text Add to dashboard Cite

Abstract-The ever-growing gap between the computation and I/O is one of the fundamental challenges for future computing systems. This computation-I/O gap is even larger for modern large scale high-performance systems due to their state-of-the-art yet decades long architecture: the compute and storage resources form two cliques that are interconnected with shared networking infrastructure. This paper presents a distributed storage middleware, called HyCache+, right on the compute nodes, which allows I/O to effectively leverage the high bi-section bandwidth of the high-speed interconnect of massively parallel high-end computing systems. HyCache+ provides the POSIX interface to end users with the memory-class I/O throughput and latency, and transparently swap the cached data with the existing slowspeed but high-capacity networked attached storage. HyCache+ has the potential to achieve both high performance and lowcost large capacity, the best of both worlds. To further improve the caching performance from the perspective of the global storage system, we propose a 2-phase mechanism to cache the hot data for parallel applications, called 2-Layer Scheduling (2LS), which minimizes the file size to be transferred between compute nodes and heuristically replaces files in the cache. We deploy HyCache+ on the IBM BlueGene/P supercomputer, and observe two orders of magnitude faster I/O throughput than the default GPFS parallel file system. Furthermore, the proposed heuristic caching approach shows 29X speedup over the traditional LRU algorithm.

show abstract

Distributed data provenance for large-scale data-intensive computing

Cited by 41 publications

References 28 publications

Virtual chunks: On supporting random accesses to scientific data in compressible storage systems

Virtual chunks: On supporting random accesses to scientific data in compressible storage systems

Efficient Scheduling of Scientific Workflows Using Hot Metadata in a Multisite Cloud

HyCache+: Towards Scalable High-Performance Caching Middleware for Parallel File Systems

Contact Info

Product

Resources

About