Provenance of Dynamic Adaptations in User-Steered Dataflows

Souza, Renan; Mattoso, Marta

doi:10.1007/978-3-319-98379-0_2

Cited by 8 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The W3C PROV ( Groth & Moreau, 2013 ) recommendation allows for a generic and uniform provenance data representation, which promotes interoperability and data analyses in general. In scientific workflows, provenance data received PROV specializations, like PROVOne ( Butt & Fitch, 2020 ) and PROV-DfA ( Souza & Mattoso, 2018 ) to cover information about workflows’ specifications, agents, activities and data derivation paths. Provenance data management requires capturing data, explicitly relating them to the workflow activities and efficiently storing these data to keep workflow high performance execution.…”

Section: Data Management In Large-scale Workflowsmentioning

confidence: 99%

Distributed in-memory data management for workflow executions

Souza

Silva

Lima

et al. 2021

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

Complex scientific experiments from various domains are typically modeled as workflows and executed on large-scale machines using a Parallel Workflow Management System (WMS). Since such executions usually last for hours or days, some WMSs provide user steering support, i.e., they allow users to run data analyses and, depending on the results, adapt the workflows at runtime. A challenge in the parallel execution control design is to manage workflow data for efficient executions while enabling user steering support. Data access for high scalability is typically transaction-oriented, while for data analysis, it is online analytical-oriented so that managing such hybrid workloads makes the challenge even harder. In this work, we present SchalaDB, an architecture with a set of design principles and techniques based on distributed in-memory data management for efficient workflow execution control and user steering. We propose a distributed data design for scalable workflow task scheduling and high availability driven by a parallel and distributed in-memory DBMS. To evaluate our proposal, we develop d-Chiron, a WMS designed according to SchalaDB’s principles. We carry out an extensive experimental evaluation on an HPC cluster with up to 960 computing cores. Among other analyses, we show that even when running data analyses for user steering, SchalaDB’s overhead is negligible for workloads composed of hundreds of concurrent tasks on shared data. Our results encourage workflow engine developers to follow a parallel and distributed data-oriented approach not only for scheduling and monitoring but also for user steering.

show abstract

Section: Data Management In Large-scale Workflowsmentioning

confidence: 99%

Distributed in-memory data management for workflow executions

Souza

Silva

Lima

et al. 2021

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, a large number of parameters and the combination of different values can easily complicate the online user steering and confuse the users if these adaptions and values are not properly monitored or registered [34]. Keeping track of each adaption (changes along with values and track of these events) as well as their order can reveal their effects on dataflow as well as workflow performance such as run-time and resource consumption [38]. This is also beneficial for users to understand what input parameters can significantly impact the results and what is the influence of specific input value for the parameters on the output [39].…”

Section: Application Of MLmentioning

confidence: 99%

Exploring the Role of Machine Learning in Scientific Workflows: Opportunities and Challenges

Nouri¹,

Davis²,

Subedi³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this survey, we discuss the challenges of executing scientific workflows as well as existing Machine Learning (ML) techniques to alleviate those challenges. We provide the context and motivation for applying ML to each step of the execution of these workflows. Furthermore, we provide recommendations on how to extend ML techniques to unresolved challenges in the execution of scientific workflows. Moreover, we discuss the possibility of using ML techniques for in-situ operations. We explore the challenges of in-situ workflows and provide suggestions for improving the performance of their execution using ML techniques.

show abstract

“…The provenance data community has significantly evolved in the recent years, oftentimes leveraging the PROV [23] family of documents, a W3C recommendation, making it a de facto standard that provides the building blocks, in terms of data representation, for any provenance-based approach, allowing for compatibility among different solutions [40]. The PROV-Wf [41] workflow provenance data representation and its derivatives [16] have also been used and evolved by several initiatives [29], [15], [42]. Our previous work builds on W3C PROV and PROV-Wf to propose PROVLake, a first provenance data representation for workflows on data lakes [9].…”

Section: Ddp 3: Provenance Of Multiple Workflows On Data Lakes Meets ...mentioning

confidence: 99%

Workflow Provenance in the Lifecycle of Scientific Machine Learning

Souza,

Azevedo,

Lourenço

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Machine Learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design principles to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the principles enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.

show abstract

Provenance of Dynamic Adaptations in User-Steered Dataflows

Cited by 8 publications

References 19 publications

Distributed in-memory data management for workflow executions

Distributed in-memory data management for workflow executions

Exploring the Role of Machine Learning in Scientific Workflows: Opportunities and Challenges

Workflow Provenance in the Lifecycle of Scientific Machine Learning

Contact Info

Product

Resources

About