Tracking and Sketching Distributed Data Provenance

Malik, Tanu; Nistor, Ligia; Gehani, Ashish

doi:10.1109/escience.2010.51

Cited by 25 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some SWf systems opt to rely on distributed file-systems that partition the metadata and store it at each node (e.g. [21], [32]), in a shared-nothing architecture, as a first step towards complete geographical distribution. Hashing is the most common technique for uniform partitioning: it consists of assigning metadata to nodes based on a hash of a file identifier.…”

Section: Related Workmentioning

confidence: 99%

Efficient Scheduling of Scientific Workflows Using Hot Metadata in a Multisite Cloud

Liu

Villaseñor-Pineda

Pacitti

et al. 2019

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Ordonnancement Efficace de Workflows Scientifiques en exploitant les Mé tadonné es Chaudes dans un Cloud Multisite Ré suméLes applications scientifiques à grande é chelle sont souvent exprimé es sous forme de workflows scientifiques (SWfs) qui aident à dé finir les jobs de traitement des donné es et les dé pendances entre les activité s des jobs. Certains SWfs né cessitent une trè s grande quantité de stockage et de calcul, ce qui peut ê tre obtenu en exploitant plusieurs data centers dans un cloud. Dans ce contexte, la gestion des mé tadonné es et l'ordonnancement des tâches entre différents data centers deviennent critiques pour l'exécution efficace de SWf. Dans cet article, nous proposons une architecture et un modè le distribué s hybrides, en utilisant les mé tadonné es chaudes (fré quemment consultées) pour l'ordonnancement efficace de SWf dans un cloud multisite. Nous utilisons notre modè le dans un systè me de gestion de workflows scientifiques (SWfMS) pour valider et ré gler son applicabilité à diffé rents workflows scientifiques ré els avec diffé rents algorithmes d'ordonnancement. Nous montrons que la combinaison d'une gestion efficace des métadonnées chaudes et des algorithmes d'ordonnancement améliore les performances du SWfMS. En é vitant les opé rations inutiles de mé tadonné es froides, le temps d'exé cution des jobs qui s'exécutent en parallèle est ré duit jusqu'à 64,1% et celui de l'ensemble des workflows scientifiques jusqu'à 37,5%. ABSTRACTLarge-scale scientific applications are often expressed as scientific workflows (SWfs) that help defining data processing jobs and dependencies between jobs' activities. Several SWfs have huge storage and computation requirements, and so they need to be processed in multiple (cloud-federated) datacenters. It has been shown that efficient metadata handling plays a key role in the performance of computing systems. However, most of this evidence concern only single-site, HPC systems to date. In addition, the efficient scheduling of tasks among different data centers is critical to the SWf execution. In this paper, we present a hybrid distributed model and architecture, using hot metadata (frequently accessed metadata) for efficient SWf scheduling in a multisite cloud. We couple our model with a scientific workflow management system (SWfMS) to validate and tune its applicability to different real-life scientific workflows with different scheduling algorithms. We show that the combination of efficient management of hot metadata and scheduling algorithms improves the performance of SWfMS, reducing the execution time of highly parallel jobs up to 64.1% and that of the whole scientific workflows up to 37.5%, by avoiding unnecessary cold metadata operations. KEYWORDShot metadata, metadata management, multisite clouds, scientific workflows, geo-distributed applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient Scheduling of Scientific Workflows Using Hot Metadata in a Multisite Cloud

Liu

Villaseñor-Pineda

Pacitti

et al. 2019

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…It is shown that the proposed approach works for the Network File System [190]. Further research extends the collection of provenance metadata to distributed systems [66,130], distributed enterprise service buses [4] and cloud services [154,155,223]. In particular, [154,155] integrate provenance into the Amazon Simple Storage Service (S3).…”

Section: Cross-system Data Flow Tracking and Policy Propagation 161mentioning

confidence: 99%

Data Usage Control for Distributed Systems

Kelbert

Pretschner

2018

ACM Trans. Priv. Secur.

View full text Add to dashboard Cite

AcknowledgementsMy thanks go to• Prof. Pretschner. For introducing me to the academic world with all its diverse aspects: administrative matters, conferences, discussions, learning, reading, presentations, projects, reviewing, teaching, writing, pain and glory. For letting me work on an interesting thesis topic. For guidance, feedback, liberty, criticism and plaudits.• Prof. Katzenbeisser. For providing feedback on earlier partial results of this thesis.• all co-authors and internal reviewers. Alexander, Dominik, Enrico, Fatemeh, Hervais, Matthias, Mojdeh, Prachi, Saahil, Sebastian, Tobias. For teaching me how to collaboratively write scientific documents and how to cope best with endless discussions, reviews, corrections, rewriting, etc.• all remaining current and former colleagues and collaborators from KIT, TUM, • all project partners from TU Darmstadt, Universität Freiburg, Universität Kassel, Capurro Fiek Stiftung für Informationsethik, Fraunhofer SIT, Google Germany GmbH, Deutsche Post AG, Nokia, IBM, AGT International, DFKI, Seeburger, Stadtwerke Saarlouis. For innumerable interesting and diverse experiences.• all developers and authors of free tools and Q&A websites. For developing and maintaining software such as Eclipse, Firefox, Gimp, Gnome, Gnuplot, Inkscape, LaTeX, LibreOffice, Linux, StackExchange, Thunderbird, and many more.• all friends and family. Which are simply too many to enumerate. For constantly reminding me, both consciously and unconsciously, in countless ways that there is more to life than work and research. AbstractData usage control provides mechanisms for data owners to remain in control over how their data is used after it has been accessed. Corresponding technical solutions are thus applicable in many distinct areas such as the protection of business, military and government secrets, intellectual property, as well as private user data. However, most existing solutions focus on the enforcement of data usage control within single systems and disregard distributed aspects of data usage control, i.e. how the usage of data can be controlled once data has been shared across systems and organizations.In fact, many data usage policies can only be enforced on a global scale, as they refer to data as well as data usage events happening within several distributed systems, e.g. "at each point in time at most two clerks might have a local copy of this contract", or "a contract must be approved by at least two clerks before it is sent to the customer".While such policies can intuitively be enforced using a centralized infrastructure, major drawbacks are that such solutions constitute a single point of failure and that they are expected to cause heavy communication and performance overheads.In order to address these open challenges, this dissertation contributes by providing (i) a formal distributed data usage control system model, and (ii) the first fully decentralized infrastructure for the preventive enforcement of data usage policies. More precisely, the provided model allows to track the f...

show abstract

“…Some workflow systems opt to rely on distributed file-systems that partition the metadata and store it at each node (e.g. [31], [32]), in a sharednothing architecture, as a first step towards complete geographical distribution. Hashing is the most common technique for uniform partitioning: it consists of assigning metadata to nodes based on a hash of a file identifier.…”

Section: Related Workmentioning

confidence: 99%

Managing hot metadata for scientific workflows on multisite clouds

Pineda-Morales

Liu

Costan

et al. 2016

2016 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Large-scale scientific applications are often expressed as workflows that help defining data dependencies between their different components. Several such workflows have huge storage and computation requirements, and so they need to be processed in multiple (cloud-federated) datacenters. It has been shown that efficient metadata handling plays a key role in the performance of computing systems. However, most of this evidence concern only single-site, HPC systems to date. In this paper, we present a hybrid decentralized/distributed model for handling hot metadata (frequently accessed metadata) in multisite architectures. We couple our model with a scientific workflow management system (SWfMS) to validate and tune its applicability to different real-life scientific scenarios. We show that efficient management of hot metadata improves the performance of SWfMS, reducing the workflow execution time up to 50% for highly parallel jobs and avoiding unnecessary cold metadata operations. Index Terms-hot metadata, metadata management, multisite clouds, scientific workflows, geo-distributed applications.

show abstract

Tracking and Sketching Distributed Data Provenance

Cited by 25 publications

References 15 publications

Efficient Scheduling of Scientific Workflows Using Hot Metadata in a Multisite Cloud

Efficient Scheduling of Scientific Workflows Using Hot Metadata in a Multisite Cloud

Data Usage Control for Distributed Systems

Managing hot metadata for scientific workflows on multisite clouds

Contact Info

Product

Resources

About