Characterizing workflow-based activity on a production e-infrastructure using provenance data

Madougou, Souley; Shahand, Shayan; Santcroos, Mark; Schaik, Barbera D.C. van; Benabdelkader, Ammar; Kampen, Antoine van; Olabarriaga, Sílvia D.

doi:10.1016/j.future.2013.04.019

Cited by 12 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ferreira et al, classifies the state of each task in a workflow and apply the appropriate rule to mitigate failures [6]; however, resource scaling were not addressed since Grid environments assign tasks in a static resource. Madougou et al, analysed task failures in an e-Infrastructure, based on history of workflow activity from the e-BioInfra platform [7]; however, the analysis only focuses on underlying resource usage and did not connect with the higher level workflow deception tasks. The work presented in [8] proposes an online mechanism for detecting anomalies while executing scientific workflows on clouds.…”

Section: Related Workmentioning

confidence: 99%

Contextual Linking between Workflow Provenance and System Performance Logs

Ahanach

Koulouzis

Zhao

2019

2019 15th International Conference on eScience (eScience)

View full text Add to dashboard Cite

When executing scientific workflows, anomalies of the workflow behavior are often caused by different issues such as resource failures at the underlying infrastructure. The provenance information collected by workflow management systems only captures the transformation of data at the workflow level. Analyzing provenance information and apposite system metrics requires expertise and manual effort. Moreover, it is often timeconsuming to aggregate this information and correlate events occurring at different levels of the infrastructure. In this paper, we propose an architecture to automate the integration among workflow provenance information and performance information from the infrastructure level. Our architecture enables workflow developers or domain scientists to effectively browse workflow execution information together with the system metrics, and analyze contextual information for possible anomalies.

show abstract

Section: Related Workmentioning

confidence: 99%

Contextual Linking between Workflow Provenance and System Performance Logs

Ahanach

Koulouzis

Zhao

2019

2019 15th International Conference on eScience (eScience)

View full text Add to dashboard Cite

show abstract

“…They categorized the virtual machine workload with respect to the following patterns: periodicity, threshold, relationship, variability, and image similarity. Madougou et al [57] provided a characterization of workflow executions using provenance data captured from a workflow management system. They analyzed usage and failure patterns at the workflow and task levels.…”

Section: Related Workmentioning

confidence: 99%

Online Task Resource Consumption Prediction for Scientific Workflows

Silva

Juve

Rynge

et al. 2015

Parallel Process. Lett.

View full text Add to dashboard Cite

Estimates of task runtime, disk space usage, and memory consumption, are commonly used by scheduling and resource provisioning algorithms to support efficient and reliable workflow executions. Such algorithms often assume that accurate estimates are available, but such estimates are difficult to generate in practice. In this work, we first profile five real scientific workflows, collecting fine-grained information such as process I/O, runtime, memory usage, and CPU utilization. We then propose a method to automatically characterize workflow task requirements based on these profiles. Our method estimates task runtime, disk space, and peak memory consumption based on the size of the tasks' input data. It looks for correlations between the parameters of a dataset, and if no correlation is found, the dataset is divided into smaller subsets using a clustering technique. Task estimates are generated based on the ratio parameter/input data size if they are correlated, or based on the probability distribution function of the parameter. We then propose an online estimation process based on the MAPE-K loop, where task executions are monitored and estimates are updated as more information becomes available. Experimental results show that our online estimation process results in much more accurate predictions than an offline approach, where all task requirements are estimated prior to workflow execution.

show abstract

“…For each workflow execution it collects data related to the jobs, their inputs and output results, users in charge of the experiments, and dependency relationships among these data. The collector follows a similar approach to our previous implementations [51,52], deploying PROV-man to gather provenance information and organize it according to the experiment context. Figure 11 illustrates two use case scenarios of the provenance collector, namely gUSE/WS-PGRADE and Neuroscience gateway where detailed information about executed workflows are gathered from gUse and NSG databases, as well as from the log files generated by the jobs executed on the DCIs.…”

Section: Provenance Of a Science Gatewaymentioning

confidence: 99%

“…Additionally, we enhanced PLIER with a set of tools to build, store, retrieve, share, and visualize workflow experiments. PLIER has been extensively used to collect and explore provenance for scientific experiments performed on a grid infrastructure, namely: (1) as an integrated component within the WS-VLAM workflow system [49,50], and (2) as a core component to automatically gather provenance data from existing grid workflow enactments services [51,52] First, we conducted a study comparing PROV to OPM, based on the provenance specifications as defined for OPM Core Specification (v1.1) and the latest PROV documentation [7]. Table 1 illustrates the main OPM concepts with their counterparts in the PROV specification.…”

Section: Introductionmentioning

confidence: 99%

PROV‐man: A PROV‐compliant toolkit for provenance management

Benabdelkader

Kampen

Olabarriaga

2015

Preprint

Self Cite

View full text Add to dashboard Cite

Discoveries in modern science can take years and involve the contribution of large amounts of data, many people and various tools. Although good scientific practice dictates that findings should be reproducible, in practice there are very few automated tools that actually support traceability of the scientific method employed, in particular when various experimental environments are involved at different research phases. Data provenance tracking approaches can play a major role in addressing many of these challenges. These approaches propose ways to capture, manage, and use of provenance information to support the traceability of the scientific methods in heterogeneous environments. PROV is a W3C standard that provides a comprensive model for data and semantics representation with common vocabularies and rich concepts to describe provenance. Nevertheless, it is difficult for domain scientists to easily understand and adopt all the richeness provided by PROV. In this paper we describe the design and implementation of the provenance manager PROV-man, a PROV-compliant framework that facilitates the tasks of scientists in integrating provenance capabilities into their data analysis tools. PROV-man provides functionalities to create and manipulate provenance data in a consistent manner and ensures its permanent storage. It also provides a set of interfaces to serialize and export provenance data into various data formats, serving interoperability. The open architecture of PROV-man, consisting of an API and a configurable database, allows for its easy deployment within existing and newly developed software tools. The paper presents examples illustrating the usage of PROV-man. The first example illustrates how to create and manipulate provenance data of an online newspaper article using PROV-man. The second example demonstrates and evaluates the PROV-man implementation in a more complex case for collection of provenance data about biomedical data analysis activities that are carried out using a distributed computing infrastructure.

show abstract

Characterizing workflow-based activity on a production e-infrastructure using provenance data

Cited by 12 publications

References 27 publications

Contextual Linking between Workflow Provenance and System Performance Logs

Contextual Linking between Workflow Provenance and System Performance Logs

Online Task Resource Consumption Prediction for Scientific Workflows

PROV‐man: A PROV‐compliant toolkit for provenance management

Contact Info

Product

Resources

About