Recording and using provenance in a protein compressibility experiment

Abstract

Very large scale computations are now becoming routinely used as a methodology to undertake scientific research. In this context, 'provenance systems' are regarded as the equivalent of the scientist's logbook for in silico experimentation: provenance captures the documentation of the process that led to some result. Using a protein compressibility analysis application, we derive a set of generic use cases for a provenance system. In order to support these, we address the following fundamental questions: what is provenance? how to record it? what is the performance impact for grid execution? what is the performance of reasoning? In doing so, we define a technologyindependent notion of provenance that captures interactions between components, internal component information and grouping of interactions, so as to allow us to analyse and reason about the execution of scientific processes. In order to support persistent provenance in heterogeneous applications, we introduce a separate provenance store, in which provenance documentation can be stored, archived and queried independently of the technology used to run the application. Through a series of practical tests, we evaluate the performance impact of such a provenance system. In summary, we demonstrate that provenance recording overhead of our prototype system remains under 10% of execution time, and we show that the recorded information successfully supports our use cases in a performant manner.

A Dataflow-Oriented Atomicity and Provenance System for Pipelined Scientific Workflows

“…Although several provenance models [7,11,2,4,10] have been proposed for scientific workflows, there has been no work on the provenance system that supports the notion of atomicity.…”

Provenance management in curated databases

“…As mentioned in the introduction, "workflow" or "coarse-grained" provenance has been studied extensively in the context of scientific computation [10,14,26]; Bose and Frew [3] and Simmhan et al [21] survey most existing research on such systems. These approaches record the process used to derive processed data products from raw data.…”

“…One form of provenance is "workflow" or "coarse-grained" provenance: information describing how derived data has been calculated from raw observations [3,10,14,21]. Workflow provenance is important in scientific computation, but is not a major concern in curated databases.…”

RDFProv: A relational RDF store for querying and managing scientific workflow provenance

“…The PReServ/PASOA system [51,52] supports the recording of interaction provenance, actor provenance, and input provenance with the provenance recording protocol, which specifies the messages that actors can asynchronously exchange with a provenance store to support provenance submission. PReServ uses a provenance management service which provides a common interface to enable different storage systems, such as file system, relational databases, XML databases, and RDF stores, as a provenance store.…”