Active Provenance for Data-Intensive Workflows: Engaging Users and Developers

Abstract

We present a practical approach for provenance capturing in Data-Intensive workflow systems. It provides contextualisation by recording injected domain metadata with the provenance stream. It offers control over lineage precision, combining automation with specified adaptations. We address provenance tasks such as extraction of domain metadata, injection of custom annotations, accuracy and integration of records from multiple independent workflows running in distributed contexts. To allow such flexibility, we introduce the concepts of programmable Provenance Types and Provenance Configuration. Provenance Types handle domain contextualisation and allow developers to model lineage patterns by redefining API methods, composing easy-to-use extensions. Provenance Configuration, instead, enables users of a Data-Intensive workflow execution o prepare it for provenance capture, by configuring the attribution of Provenance Types to components and by specifying grouping into semantic clusters. This enables better searches over the lineage records. Provenance Types and Provenance Configuration are demonstrated in a system being used by computational seismologists. It is based on an extended provenance model, S-PROV.

Comprehensible Control for Researchers and Developers Facing Data Challenges

“…In complex collaborations their work interacts with that of other experts, and extends into new territory. A comprehensive provenance system facilitates this [6]. It enables them to see what is happening, wherever it is happening.…”

“…Provenance provides definitive evidence of what has happened. P4 extends the VERCE system [2] with improved precision, controls on provenance capture, domain metadata injection, agile use, tools and visualisations with steerable clustering [13], [6]. These extend the W3C-PROV standard 5 and the ProvONE developments 6 .…”

“…P4 extends the VERCE system [2] with improved precision, controls on provenance capture, domain metadata injection, agile use, tools and visualisations with steerable clustering [13], [6]. These extend the W3C-PROV standard 5 and the ProvONE developments 6 . DARE is extending the collection and control mechanisms in P4 to make provenance pervasive and protected, i.e., captured for every significant action, granting researchers control on disclosing and sharing their traces.…”

“…We outline the provenance functionality, described in [6], that enables the acquisition and exploitation of provenance data. We capture users' choices when they customise their computational space, track reproducible progress and share methods and results within or beyond DARE.…”

Towards a Computer-Interpretable Actionable Formal Model to Encode Data Governance Rules

“…An explicit direction may be to define a standard protocol to retrieve rules with data; real-time processing (instead of the current retrospective analysis) should also be investigated. This would enable alerts to be sent to users when new obligations emerge and eventually lead to warnings preventing serious rule infringement, exploiting active provenance [27].…”