Kepler/pPOD: Scientific Workflow and Provenance Support for Assembling the Tree of Life

Bowers, Shawn; McPhillips, Timothy M.; Riddle, Sean; Anand, Manish Kumar; Ludäscher, Bertram

doi:10.1007/978-3-540-89965-5_9

Cited by 47 publications

(40 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite treating the representation of parameter values, AWARD does not handle parameter values as data elements in its relational representation, which prevents the management of the dataflow at the logical level. There are some SWMS with dataflow management support at the logical level, such as Kepler [24], Panda [25] and Chiron [12]. SWMS typically register dataflow as workflow provenance.…”

Section: Raw Data and Workflowsmentioning

confidence: 99%

Raw data queries during data-intensive parallel workflow execution

Silva

Leite

Camata

et al. 2017

Future Generation Computer Systems

View full text Add to dashboard Cite

Computer simulations consume and produce huge amounts of raw data files presented in different formats, e.g., HDF5 in computational fluid dynamics simulations. Users often need to analyze domain-specific data based on related data elements from multiple files during the execution of computer simulations. In a raw data analysis, one should identify regions of interest in the data space and retrieve the content of specific related raw data files. Existing solutions, such as FastBit and RAW, are limited to a single raw data file analysis and can only be used after the execution of computer simulations. Scientific Workflow Management Systems (SWMS) can manage the dataflow of computer simulations and register related raw data files at a provenance database. This paper aims to combine the advantages of a dataflow-aware SWMS and the raw data file analysis techniques to allow for queries on raw data file elements that are related, but reside in separate files. We propose a component-based architecture, named as ARMFUL (Analysis of Raw data from Multiple Files) with raw data extraction and indexing techniques, which allows for a direct access to specific elements or regions of raw data space. ARMFUL innovates by using a SWMS provenance database to add a dataflow access path to raw data files. ARMFUL facilitates the invocation of ad-hoc programs and third party tools (e.g., FastBit tool) for raw data analyses. In our experiments, a real parallel computational fluid dynamics is executed, exploring different alternatives of raw data extraction, indexing and analysis.

show abstract

Section: Raw Data and Workflowsmentioning

confidence: 99%

Raw data queries during data-intensive parallel workflow execution

Silva

Leite

Camata

et al. 2017

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…Scientific workflow systems are being used in many scientific domains, and many approaches have been proposed recently for representing and storing workflow provenance [5,6]. However, most of the existing provenance approaches store provenance for a single runs, and do not capture or maintain associations across runs [22,23,24,25]. The framework described in [7] records associations between multiple related workflow runs.…”

Section: Related Workmentioning

confidence: 99%

Understanding Collaborative Studies through Interoperable Workflow Provenance

Altıntaş

Anand

Crawl

et al. 2010

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. The provenance of a data product contains information about how the product was derived, and is crucial for enabling scientists to easily understand, reproduce, and verify scientific results. Currently, most provenance models are designed to capture the provenance related to a single run, and mostly executed by a single user. However, a scientific discovery is often the result of methodical execution of many scientific workflows with many datasets produced at different times by one or more users. Further, to promote and facilitate exchange of information between multiple workflow systems supporting provenance, the Open Provenance Model (OPM) has been proposed by the scientific workflow community. In this paper, we describe a new query model that captures implicit user collaborations. We show how this model maps to OPM and helps to answer collaborative queries, e.g., identifying combined workflows and contributions of users collaborating on a project based on the records of previous workflow executions. We also adopt and extend the high-level Query Language for Provenance (QLP) with additional constructs, and show how these extensions allow non-expert users to express collaborative provenance queries against this model easily and concisely. Furthermore, we adopt the Provenance Challenge 3 (PC3) workflows as a collaborative and interoperable usecase scenario, where different stages of the workflow are executed in three different workflow environmentsKepler, Taverna, and WSVLAM. Through this usecase, we demonstrate how we can establish and understand collaborative studies through interoperable workflow provenance.

show abstract

“…Provenance techniques have since been studied in the context of databases [9,29], scientific workflow systems [43,7], operating systems [48], and inference systems [38] (including recent interest in the Semantic Web community, culminating in a W3C Working Group on Provenance [27,47]). In each of these contexts, there is a large design space for provenance mechanisms, yet at the same time there is not a clear consensus on the requirements or policies that these mechanisms ought to satisfy.…”

Section: Introductionmentioning

confidence: 99%

Toward a Theory of Self-explaining Computation

Cheney

Acar

Perera

2013

In Search of Elegance in the Theory and Practice of Computation

View full text Add to dashboard Cite

Abstract. Provenance techniques aim to increase the reliability of human judgments about data by making its origin and derivation process explicit. Originally motivated by the needs of scientific databases and scientific computation, provenance has also become a major issue for business and government data on the Web. However, so far provenance has been studied only in relatively restrictive settings: typically, for data stored in databases or scientific workflow systems, and processed by query or workflow languages of limited expressiveness. Long-term provenance solutions require an understanding of provenance in other settings, particularly the general-purpose programming or scripting languages that are used to glue different components such as databases, Web services and workflows together. Moreover, what is required is not only an account of mechanisms for recording provenance, but also a theory of what it means for provenance information to explain or justify a computation. In this paper, we begin to outline a such a theory of self-explaining computation. We introduce a model of provenance for a simple imperative language based on operational derivations and explore its properties.

show abstract

Kepler/pPOD: Scientific Workflow and Provenance Support for Assembling the Tree of Life

Cited by 47 publications

References 11 publications

Raw data queries during data-intensive parallel workflow execution

Raw data queries during data-intensive parallel workflow execution

Understanding Collaborative Studies through Interoperable Workflow Provenance

Toward a Theory of Self-explaining Computation

Contact Info

Product

Resources

About