Differencing Provenance in Scientific Workflows

Bao, Zhuowei; Cohen-Boulakia, Sarah; Davidson, Susan B.; Eyal, Anat; Khanna, Sanjeev

doi:10.1109/icde.2009.103

Cited by 47 publications

(35 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Our approach allows users to share their experiments in the spirit of [39], understand and compare their results [40] and possibly refine their analysis process to augment quality of their data sets. To do so, we have followed the recommendations and current standards on provenance [41] [19] and introduced a generator of IPython/Jupyter notebooks [2].…”

Section: Discussionmentioning

confidence: 99%

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

Pradal

Artzet

Chopard

et al. 2017

Future Generation Computer Systems

Self Cite

View full text Add to dashboard Cite

Plant phenotyping consists in the observation of physical and biochemical traits of plant genotypes in response to environmental conditions. Challenges, in particular in context of climate change and food security, are numerous. High-throughput platforms have been introduced to observe the dynamic growth of a large number of plants in different environmental conditions. Instead of considering a few genotypes at a time (as it is the case when phenomic traits are measured manually), such platforms make it possible to use completely new kinds of approaches. However, the data sets produced by such widely instrumented platforms are huge, constantly augmenting and produced by increasingly complex experiments, reaching a point where distributed computation is mandatory to extract knowledge from data.In this paper, we introduce InfraPhenoGrid, the infrastructure we designed and deploy to efficiently manage data sets produced by the PhenoArch plant phenomics platform in the context of the French Phenome Project. Our solution consists in deploying scientific workflows on a Grid using a middleware to pilot workflow executions. Our approach is user-friendly in the sense that despite the intrinsic complexity of the infrastructure, running scientific workflows and understanding results obtained (using provenance information) is kept as simple as possible for end-users.

show abstract

Section: Discussionmentioning

confidence: 99%

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

Pradal

Artzet

Chopard

et al. 2017

Future Generation Computer Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Answering these questions requires (inductive or recursive) querying along dependency paths of unknown length, making them computationally expensive [10], [7], [13] as they involve the transitive closure of data dependencies ddep. A particular challenge of our collaborative e-Science scenario is to ensure that the traces (e.g., T A and T B in Figs.…”

Section: B Queries On the Provenance Modelmentioning

confidence: 99%

“…Our model is similar to the one in [7] in that they also use a graph homomorphism to define when a trace (called run there) T is valid w.r.t. a given workflow graph W .…”

Section: A Abstract Workflow and Provenance Modelmentioning

confidence: 99%

See 1 more Smart Citation

Linking multiple workflow provenance traces for interoperable collaborative science

Missier

Ludäscher

Bowers

et al. 2010

The 5th Workshop on Workflows in Support of Large-Scale Science

View full text Add to dashboard Cite

Abstract-Scientific collaboration increasingly involves data sharing between separate groups. We consider a scenario where data products of scientific workflows are published and then used by other researchers as inputs to their workflows. For proper interpretation, shared data must be complemented by descriptive metadata. We focus on provenance traces, a prime example of such metadata which describes the genesis and processing history of data products in terms of the computational workflow steps. Through the reuse of published data, virtual, implicitly collaborative experiments emerge, making it desirable to compose the independently generated traces into global ones that describe the combined executions as single, seamless experiments. We present a model for provenance sharing that realizes this holistic view by overcoming the various interoperability problems that emerge from the heterogeneity of workflow systems, data formats, and provenance models. At the heart lie (i) an abstract workflow and provenance model in which (ii) data sharing becomes itself part of the combined workflow. We then describe an implementation of our model that we developed in the context of the Data Observation Network for Earth (DataONE) project and that can "stitch together" traces from different Kepler and Taverna workflow runs. It provides a prototypical framework for seamless cross-system, collaborative provenance management and can be easily extended to include other systems. Our approach also opens the door to new ways of workflow interoperability not only through often elusive workflow standards but through shared provenance information from public repositories.

show abstract

“…Bao et al [15] introduce an algorithm for differencing provenance (due to workflow execution). The difference or edit distance between a pair of valid runs of the same specification is defined as a minimum cost sequence of edit operations that transform one run to the other.…”

Section: Querying Provenancementioning

confidence: 99%

The Foundations for Provenance on the Web

Moreau

2010

FNT in Web Science

172

View full text Add to dashboard Cite

Provenance, i.e., the origin or source of something, is becoming an important concern, since it offers the means to verify data products, to infer their quality, to analyse the processes that led to them, and to decide whether they can be trusted. For instance, provenance enables the reproducibility of scientific results; provenance is necessary to track attribution and credit in curated databases; and, it is essential for reasoners to make trust judgements about the information they use over the Semantic Web.As the Web allows information sharing, discovery, aggregation, filtering and flow in an unprecedented manner, it also becomes very difficult to identify, reliably, the original source that produced an information item on the Web. Since the emerging use of provenance in niche applications is undoubtedly demonstrating the benefits of provenance, we contend that provenance can and should reliably be tracked and exploited on the Web, and we survey the necessary foundations to achieve such a vision.Using multiple data sources, we have compiled the largest bibliographical database on provenance so far. This large corpus allows us to analyse emerging trends in the research community. Specifically, using the CiteSpace tool, we identify clusters of papers that constitute research fronts, from which we derive characteristics that we use to structure our foundational framework for provenance on the Web. We note that such an endeavour requires a multi-disciplinary approach, since it requires contributions from many computer science sub-disciplines, but also other non-technical fields given the human challenge that is anticipated.To develop our vision, it is necessary to provide a definition of provenance that applies to the Web context. Our conceptual definition of provenance is expressed in terms of processes, and is shown to generalise various definitions of provenance commonly encountered. Furthermore, by bringing realistic distributed systems assumptions, we refine our definition as a query over assertions made by processes.Given that the majority of work on provenance has been undertaken by the database, workflow and e-science communities, we review some of their work, contrasting approaches, and focusing on important topics we believe to be crucial for bringing provenance to the Web, such as abstraction, collections, storage, queries, workflow evolution, semantics and activities involving human interactions.However, provenance approaches developed in the context of databases and workflows essentially deal with closed systems. By that, we mean that workflow or database management systems are in full control of the data they manage, and track their provenance within their own scope, but not beyond. In the context of the Web, a broader approach is required by which chunks of provenance representation can be brought together to describe the provenance of information flowing across multiple systems. This is the specific purpose of the Open Provenance Vision, which is an approach that consists of controlled voc...

show abstract

Differencing Provenance in Scientific Workflows

Cited by 47 publications

References 20 publications

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid

Linking multiple workflow provenance traces for interoperable collaborative science

The Foundations for Provenance on the Web

Contact Info

Product

Resources

About