Connecting Scientific Data to Scientific Experiments with Provenance

Miles, Simon; Deelman, Ewa; Groth, Paul; Vahi, Karan; Mehta, Gaurang; Moreau, Luc

doi:10.1109/e-science.2007.22

Cited by 27 publications

(31 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is then stored in the user's repository, although it does not enable other users of the data to determine its provenance. The Provenance Aware Service Oriented Architecture (PASOA) project [41] arranges for data transformations to be reported to a central provenance service [60], which can be queried by other users as well. The more recent ES3 model [13] extracts provenance information automatically from arbitrary applications by monitoring their interactions with their execution environment and logs them to a customized database.…”

Section: Related Workmentioning

confidence: 99%

SPADE: Support for Provenance Auditing in Distributed Environments

Gehani

Tariq

2012

Lecture Notes in Computer Science

177

138

View full text Add to dashboard Cite

Abstract. SPADE is an open source software infrastructure for data provenance collection and management. The underlying data model used throughout the system is graph-based, consisting of vertices and directed edges that are modeled after the node and relationship types described in the Open Provenance Model. The system has been designed to decouple the collection, storage, and querying of provenance metadata. At its core is a novel provenance kernel that mediates between the producers and consumers of provenance information, and handles the persistent storage of records. It operates as a service, peering with remote instances to enable distributed provenance queries. The provenance kernel on each host handles the buffering, filtering, and multiplexing of incoming metadata from multiple sources, including the operating system, applications, and manual curation. Provenance elements can be located locally with queries that use wildcard, fuzzy, proximity, range, and Boolean operators. Ancestor and descendant queries are transparently propagated across hosts until a terminating expression is satisfied, while distributed path queries are accelerated with provenance sketches.

show abstract

Section: Related Workmentioning

confidence: 99%

SPADE: Support for Provenance Auditing in Distributed Environments

Gehani

Tariq

2012

Lecture Notes in Computer Science

177

138

View full text Add to dashboard Cite

show abstract

“…Examples of such systems include operating system [218][159], desktop [265], statistical packages [143,21], databases [59,406,176], data warehouses [104], and workflow systems (Kepler [39,41], Taverna [419], VisTrails [156], VDS [424], Pegasus [278]). Other approaches allow for varying degrees of open-ness: curated databases allow for user-edits [52], PASOA allows for service-oriented architectures [195] where the number of components can dynamically change and is not known at design-time, or some allow for data to be published on the Web, with data and provenance both accessible by simple browser navigation [447].…”

Section: Assumptionsmentioning

confidence: 99%

“…Hence, a primary driver for provenance is reproducibility of scientific analyses and processes. Furthemore, provenance is not only used for interpreting data and providing reproducible results, but also for troubleshooting and optimization [278]. For an extensive analysis of user requirements for provenance in e-Science, we refer the reader to [279].…”

Section: Chapter 4 Provenance In Workflows and Databasesmentioning

confidence: 99%

“…The Pegasus workflow compiler [278] takes as input an abstract workflow (specified as a DAG) that is compiled into an instantiated workflow, directly executable by a workflow engine, where computation location, data transfers and libraries to invoke have all been made explicit. In this case, provenance can be used for troubleshooting and understanding runtime behaviour of the workflow system and of the application; this is achieved by connecting runtime information to the original abstract specification, as designed by the user.…”

Section: Workflow Evolutionmentioning

confidence: 99%

See 1 more Smart Citation

The Foundations for Provenance on the Web

Moreau

2010

FNT in Web Science

Self Cite

172

View full text Add to dashboard Cite

Provenance, i.e., the origin or source of something, is becoming an important concern, since it offers the means to verify data products, to infer their quality, to analyse the processes that led to them, and to decide whether they can be trusted. For instance, provenance enables the reproducibility of scientific results; provenance is necessary to track attribution and credit in curated databases; and, it is essential for reasoners to make trust judgements about the information they use over the Semantic Web.As the Web allows information sharing, discovery, aggregation, filtering and flow in an unprecedented manner, it also becomes very difficult to identify, reliably, the original source that produced an information item on the Web. Since the emerging use of provenance in niche applications is undoubtedly demonstrating the benefits of provenance, we contend that provenance can and should reliably be tracked and exploited on the Web, and we survey the necessary foundations to achieve such a vision.Using multiple data sources, we have compiled the largest bibliographical database on provenance so far. This large corpus allows us to analyse emerging trends in the research community. Specifically, using the CiteSpace tool, we identify clusters of papers that constitute research fronts, from which we derive characteristics that we use to structure our foundational framework for provenance on the Web. We note that such an endeavour requires a multi-disciplinary approach, since it requires contributions from many computer science sub-disciplines, but also other non-technical fields given the human challenge that is anticipated.To develop our vision, it is necessary to provide a definition of provenance that applies to the Web context. Our conceptual definition of provenance is expressed in terms of processes, and is shown to generalise various definitions of provenance commonly encountered. Furthermore, by bringing realistic distributed systems assumptions, we refine our definition as a query over assertions made by processes.Given that the majority of work on provenance has been undertaken by the database, workflow and e-science communities, we review some of their work, contrasting approaches, and focusing on important topics we believe to be crucial for bringing provenance to the Web, such as abstraction, collections, storage, queries, workflow evolution, semantics and activities involving human interactions.However, provenance approaches developed in the context of databases and workflows essentially deal with closed systems. By that, we mean that workflow or database management systems are in full control of the data they manage, and track their provenance within their own scope, but not beyond. In the context of the Web, a broader approach is required by which chunks of provenance representation can be brought together to describe the provenance of information flowing across multiple systems. This is the specific purpose of the Open Provenance Vision, which is an approach that consists of controlled voc...

show abstract

“…This is demonstrated by the numerous research and industrial initiatives to develop "provenance-aware" systems [41,34,16,15,44,27]. Of particular interest are applications in which the data and workflow are distributed among several heterogeneous and autonomous information systems and are often combined to make useful analysis.…”

Section: Introductionmentioning

confidence: 99%

Efficient querying of distributed provenance stores

Gehani

Kim

Malik

2010

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

View full text Add to dashboard Cite

Current projects that automate the collection of provenance information use a centralized architecture for managing the resulting metadata -that is, provenance is gathered at remote hosts and submitted to a central provenance management service. In contrast, we are developing a completely decentralized system with each computer maintaining the authoritative repository of the provenance gathered on it. Our model has several advantages, such as scaling to large amounts of metadata generation, providing low-latency access to provenance metadata about local data, avoiding the need for synchronization with a central service after operating while disconnected from the network, and letting users retain control over their data provenance records. We describe the SPADE project's support for tracking data provenance in distributed environments, including how queries can be optimized with provenance sketches, pre-caching, and caching.

show abstract

Connecting Scientific Data to Scientific Experiments with Provenance

Abstract: Abstract

Cited by 27 publications

References 18 publications

SPADE: Support for Provenance Auditing in Distributed Environments

SPADE: Support for Provenance Auditing in Distributed Environments

The Foundations for Provenance on the Web

Efficient querying of distributed provenance stores

Contact Info

Product

Resources

About