Cloud infrastructure provenance collection and management to reproduce scientific workflows execution

Abstract

The emergence of Cloud computing provides a new computing paradigm for scientific workflow execution. It provides dynamic, on-demand and scal-able resources that enable the processing of complex workflow-based experiments. With the ever growing size of the experimental data and increasingly complex processing workflows, the need for reproducibility has also become essential. Provenance has been thought of a mechanism to verify a workflow and to provide workflow reproducibility. One of the obstacles in reproduc-ing an experiment execution is the lack of information about the execution infrastructure in the collected provenance. This information becomes crit-ical in the context of Cloud in which resources are provisioned on-demand and by specifying resource configurations. Therefore, a mechanism is re-quired that enables capturing of infrastructure information along with the provenance of workflows executing on the Cloud to facilitate the re-creation of execution environment on the Cloud. This paper presents a framework, ReCAP, along with the proposed mapping approaches that aid in captur-ing the Cloudaware provenance information and help in re-provisioning the execution resource on the Cloud with similar configurations. Experimental evaluation has shown the impact of different resource configurations on the workflow execution performance, therefore justifies the need for collecting such provenance information in the context of Cloud. The evaluation has also demonstrated that the proposed mapping approaches can capture Cloud information in various Cloud usage scenarios without causing performance overhead and can also enable the re-provisioning of resources on Cloud. Experiments were conducted using workflows from different scientific domains such as astronomy and neuroscience to demonstrate the applicability of this research for different workflows.

A Serverless Tool for Platform Agnostic Computational Experiment Management

“…In LONI pipeline, a provenance model exists which includes detailed records of data use and file lifecycle ( Dinov et al, 2010 ), which is designed to inform data consumers what types of analyses can be and have been performed with the data in question; this tool is tightly coupled with the LONI pipeline ecosystem. The ReCAP ( Hasham et al, 2018 ) project has been developed to evaluate the resource consumption of arbitrary pipelines on the cloud and can aid in cloud-instance optimization. While this tool has potential for a large impact in designing both cost effective and scalable analyses, there is considerable overhead as it manages executions through a persistent server and workflow engine.…”

“…These systems deploy tools on HPC environments and record detailed execution information so that scientists can keep accurate records and debug their workflows. Tools such as LONI’s provenance manager (Dinov et al, 2010), Reprozip (Chirigati et al, 2016), and ReCAP (Hasham et al, 2018) capture system-level properties such as system resources consumed and files accessed, where tools supporting the Neuroimaging Data Model (NIDM) (Sochat and Nichols, 2016), a neuroimaging-specific provenance model based on W3C-PROV (Missier et al, 2013), capture information about the domain-specific transformations applied to the data of interest.…”

A novel approach to provenance management for privacy preservation

“…The framework aims to protect data privacy from storage servers. In Hasham et al [18], another framework is presented and cloud usage scenarios are studied for the execution of scientific workflows with cloud infrastructure in order to generate the cloud-aware provenance.…”

Editorial: New developments in cloud and IoT

“…The paper by Hasham et al [10] illustrates benefits (such as dynamic, on-demand and scalable provisioning of resources) that cloud computing provides for the execution of scientific workflows. Accordingly, it presents a framework that aims to reproduces Scientific Workflow Execution using Cloud-Aware Provenance (ReCAP).…”