Data reduction in scientific workflows using provenance monitoring and user steering

Souza, Renan; Silva, Vítor; Coutinho, Álvaro L. G. A.; Valduriez, Patrick; Mattoso, Marta

doi:10.1016/j.future.2017.11.028

Cited by 14 publications

(31 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we briefly explain each of them. Although presented separately, we notice that those categories share a lot of data (Ogasawara et al, 2011;Oliveira et al, 2010;Silva et al, 2017;Souza et al, 2020). Storing them separately leads to data redundancy and lack of data integration support for runtime data analysis.…”

Section: Data Management In Large-scale Workflowsmentioning

confidence: 99%

“…Risers Fatigue Analysis Workflow ( Souza et al, 2020 ) is a real case study from the Oil & Gas industry. This workflow calculates the fatigue of ultra-deep oil platform structures, such as risers.…”

Section: Experimental Evaluationmentioning

confidence: 99%

“…In our past work with Chiron WMS ( Ogasawara et al, 2011 ; Ogasawara et al, 2013 ), we have shown how users can have a better understanding of the data being processed in their experiments while the workflows are executing ( Silva et al, 2017 ; Silva et al, 2020 ). Then, based on such understanding, users may decide to dynamically adapt the workflow, such as reduce datasets ( Souza et al, 2020 ), change user-defined loop conditions ( Dias et al, 2015 ), and parameters ( Camata et al, 2018 ; Souza et al, 2019 ). Chiron adopts an integrated data management solution to store data dependencies, execution data, domain data, and provenance data all together in the same database available for monitoring and user steering.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Distributed in-memory data management for workflow executions

Souza

Silva

Lima

et al. 2021

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

Complex scientific experiments from various domains are typically modeled as workflows and executed on large-scale machines using a Parallel Workflow Management System (WMS). Since such executions usually last for hours or days, some WMSs provide user steering support, i.e., they allow users to run data analyses and, depending on the results, adapt the workflows at runtime. A challenge in the parallel execution control design is to manage workflow data for efficient executions while enabling user steering support. Data access for high scalability is typically transaction-oriented, while for data analysis, it is online analytical-oriented so that managing such hybrid workloads makes the challenge even harder. In this work, we present SchalaDB, an architecture with a set of design principles and techniques based on distributed in-memory data management for efficient workflow execution control and user steering. We propose a distributed data design for scalable workflow task scheduling and high availability driven by a parallel and distributed in-memory DBMS. To evaluate our proposal, we develop d-Chiron, a WMS designed according to SchalaDB’s principles. We carry out an extensive experimental evaluation on an HPC cluster with up to 960 computing cores. Among other analyses, we show that even when running data analyses for user steering, SchalaDB’s overhead is negligible for workloads composed of hundreds of concurrent tasks on shared data. Our results encourage workflow engine developers to follow a parallel and distributed data-oriented approach not only for scheduling and monitoring but also for user steering.

show abstract

Section: Data Management In Large-scale Workflowsmentioning

confidence: 99%

Section: Experimental Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Distributed in-memory data management for workflow executions

Souza

Silva

Lima

et al. 2021

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…All information are stored in a workflow database and thus can be changed during run time. In [26], this feature is used to reduce the data while it is already processed. However, instead of removing, data can also be added.…”

Section: Related Workmentioning

confidence: 99%

Executing cyclic scientific workflows in the cloud

2021

View full text Add to dashboard Cite

We present an algorithm and a software architecture for a cloud-based system that executes cyclic scientific workflows whose structure may change during run time. Existing approaches either rely on workflow definitions based on directed acyclic graphs (DAGs) or require workarounds to implement cyclic structures. In contrast, our system supports cycles natively, avoids workarounds, and as such reduces the complexity of workflow modelling and maintenance. Our algorithm traverses workflow graphs and transforms them iteratively into linear sequences of executable actions. We call these sequences process chains. Our software architecture distributes the process chains to multiple compute nodes in the cloud and oversees their execution. We evaluate our approach by applying it to two practical use cases from the domains of astronomy and engineering. We also compare it with two existing workflow management systems. The evaluation demonstrates that our algorithm is able to execute dynamically changing workflows with cycles and that design and maintenance of complex workflows is easier than with existing solutions. It also shows that our software architecture can run process chains on multiple compute nodes in parallel to significantly speed up the workflow execution. An implementation of our algorithm and the software architecture is available with the Steep Workflow Management System that we released under an open-source license. The resources for the first practical use case are also available as open source for reproduction.

show abstract

“…In Davidson et al [11], a theoretical model of privacy for module functionality in workflows is developed. An adaptive workflow monitoring approach that proposes a solution to the input data reduction problem in scientific workflows is presented in Souza et al [12]. Morshedzadeh et al [13] present a PMS that is based on an industrial case study for the product lifecycle management.…”

Section: Related Workmentioning

confidence: 99%

A novel approach to provenance management for privacy preservation

Can

Yilmazer

2019

Journal of Information Science

View full text Add to dashboard Cite

Provenance determines the origin of the data by tracing and recording the actions that are performed on the data. Therefore, provenance is used in many fields to ensure the reliability and quality of data. In this work, provenance information is used to meet the security needs in information systems. For this purpose, a domain-independent provenance model is proposed. The proposed provenance model is based on the Open Provenance Model and Semantic Web technologies. The goal of the proposed provenance model is to integrate the provenance and security concepts in order to detect privacy violations by querying the provenance data. In order to evaluate the proposed provenance model, we illustrated our domain-independent model by integrating it with an infectious disease domain and implemented the Healthcare Provenance Information System.

show abstract

Data reduction in scientific workflows using provenance monitoring and user steering

Cited by 14 publications

References 30 publications

Distributed in-memory data management for workflow executions

Distributed in-memory data management for workflow executions

Executing cyclic scientific workflows in the cloud

A novel approach to provenance management for privacy preservation

Contact Info

Product

Resources

About