ProvManager: a provenance management system for scientific workflows

Abstract

Running scientific workflows in distributed and heterogeneous environments has been a motivating approach for provenance management, which is loosely coupled to the workflow execution engine. This kind of approach is interesting because it allows both storage and access to provenance data in a homogeneous way, even in an environment where different workflow management systems work together. However, current approaches overload scientists with many ad hoc tasks, such as script adaptations and implementations of extra functionalities to provide provenance independence. This paper proposes ProvManager, a provenance management approach that eases the gathering, storage, and analysis of provenance information in a distributed and heterogeneous environment scenario, without putting the burden of adaptations on the scientist. ProvManager leverages the provenance management at the experiment level by integrating different workflow executions from multiple workflow management systems.

Implicit provenance gathering through configuration management

“…The ProvManager [3] approach, as previously discussed, works by adapting workflow activities. It minimizes the overhead of activities instrumentation via an automatic adaptation process.…”

“…This provenance gathering mechanism uses Git [9] VCS to capture and store retrospective provenance, including ImP. The ProvMonitor implementation is integrated with ProvManager [3], an activity-based provenance gathering mechanism that works through automatic workflow instrumentation. This way, each activity becomes responsible for gathering its own provenance.…”

Scientific Workflow Interchanging through Patterns: Reversals and Lessons Learned

“…These two solutions are nevertheless limited to these two systems and their adaptability to a broader range of SWfMSs would depend on a complete reformulation of their architectures. Other pieces of work [19], [20] propose approaches to the interoperation of different SWfMSs based on provenance metadata stored according to the OPM [21] and PROV [22] standards. The achieved interoperation in these solutions is retrospective though, in the sense they capture metadata about past workflow executions from different SWfMSs and allow for joint analyses over them; therefore, they do not give support for scientists to reuse workflow specifications across these different SWfMSs.…”

A Provenance-Aware Access Control Framework with Typed Provenance

“…The observed strategy requires operating systems to continuously collect provenance about running processes, their inputs and outputs [19]. The disclosed strategy requires adapted applications to collect provenance as designed by software architects [20], [21]. Users sometimes need to manually declare provenance when it cannot be captured by the system or application [11], [18].…”

“…A query engine is usually specific to a storage model. Hence, users have to write queries in languages specific to the storage model, such as SQL [22], Prolog [21], or SPARQL [23]. However, these general languages were not designed specifically for provenance.…”