Provenance Annotation and Analysis to Support Process Re-computation

Abstract

Many resource-intensive analytics processes evolve over time following new versions of the reference datasets and software dependencies they use. We focus on scenarios in which any version change has the potential to affect many outcomes, as is the case for instance in high throughput genomics where the same process is used to analyse large cohorts of patient genomes, or cases. As any version change is unlikely to affect the entire population, an efficient strategy for restoring the currency of the outcomes requires first to identify the scope of a change, i.e., the subset of affected data products. In this paper we describe a generic and reusable provenance-based approach to address this scope discovery problem. It applies to a scenario where the process consists of complex hierarchical components, where different input cases are processed using different version configurations of each component, and where separate provenance traces are collected for the executions of each of the components. We show how a new data structure, called a restart tree, is computed and exploited to manage the change scope discovery problem.

Science Capsule: Towards Sharing and Reproducibility of Scientific Workflows

“…However, these tools require adapting existing user workflows and work practices to a specific software ecosystem, which is a challenge for scientific ecosystems. Also, they do not extensively capture the human elements of a workflow (e.g., lab notebooks), and are primarily limited to simple user annotations [11], [12]. Science Capsule collects workflow runtime and provenance information in a non-intrusive manner through system-level monitoring, while allowing users to enhance the information by managing userdefined artifacts as part of the workflow.…”

“…This is challenging for workflows using experimental and observational data like that from the light sources, where researchers reuse workflows for real-time data analyses to refine the experiment and for post-experiment data analyses. Previous work in recording human activities in provenance is also limited to simple annotations [11], [12] that is not sufficient to capture the scientific complexities and thought process behind an experiment design or a workflow. Thus, there is a need to capture the relevant context about a workflow and its artifacts along with runtime information; that can seamlessly integrate with existing scientific software ecosystems to enable sharing, reuse and reproduction of workflows and their data.…”