Managing Event Oriented Workflows

Marchant, David; Munk, Rasmus; Brenne, Elise Otterlei; Vinter, Brian

doi:10.1109/xloop51963.2020.00009

Cited by 5 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Provenance databases share a lot of information used in task scheduling like tasks, task parameter values, task input data values, dependencies, and task execution time. These information allow for analyses of task execution time, detecting outliers, in addition to a task execution derivation path with its associated input and parameter data [13,26,29].…”

Section: Related Workmentioning

confidence: 99%

“…Despite solutions for self-tuning based on machine learning [40], decisions like changing convergence values, the number of iterations, or levels of interpolation still need human interference, which complements AI-based solutions [19,52]. Supporting user steering in scientific experiments allows users to run data analyses at runtime (e.g., inspect, debug, visualize, monitor) that may lead to dynamic adaptation of aspects of the workflow (e.g., change the input data, parameters, convergence criteria) [6,17,26,29,30].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Distributed in-memory data management for workflow executions

Souza

Silva

Lima

et al. 2021

PeerJ Computer Science

View full text Add to dashboard Cite

Complex scientific experiments from various domains are typically modeled as workflows and executed on large-scale machines using a Parallel Workflow Management System (WMS). Since such executions usually last for hours or days, some WMSs provide user steering support, i.e., they allow users to run data analyses and, depending on the results, adapt the workflows at runtime. A challenge in the parallel execution control design is to manage workflow data for efficient executions while enabling user steering support. Data access for high scalability is typically transaction-oriented, while for data analysis, it is online analytical-oriented so that managing such hybrid workloads makes the challenge even harder. In this work, we present SchalaDB, an architecture with a set of design principles and techniques based on distributed in-memory data management for efficient workflow execution control and user steering. We propose a distributed data design for scalable workflow task scheduling and high availability driven by a parallel and distributed in-memory DBMS. To evaluate our proposal, we develop d-Chiron, a WMS designed according to SchalaDB’s principles. We carry out an extensive experimental evaluation on an HPC cluster with up to 960 computing cores. Among other analyses, we show that even when running data analyses for user steering, SchalaDB’s overhead is negligible for workloads composed of hundreds of concurrent tasks on shared data. Our results encourage workflow engine developers to follow a parallel and distributed data-oriented approach not only for scheduling and monitoring but also for user steering.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Distributed in-memory data management for workflow executions

Souza

Silva

Lima

et al. 2021

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…A final test was conducted using the scientific analysis first presented in [15]. This is an example of a real-world scientific workflow, and is tested here to show that our results above still apply to actual analysis.…”

Section: Actual Scientific Work Testmentioning

confidence: 99%

“…A more complex rules based systems is Managing Event Oriented Workflows(MEOW) [15]. This is a structure for defining rules which can match broad categories of events and can schedule arbitrary analysis in response.…”

Section: Introductionmentioning

confidence: 99%

Slurm Scheduling From Rules-Based Systems

Blomqvist,

Marchant

Kalpa Publications in Computing

View full text Add to dashboard Cite

This paper explores how a rules based scheduling system can be integrated with a traditional workload manager such as slurm. This integration will be done with as minimal additional setup by a user as is feasible, while maintaining security of the network, as well as any machines involved. To this effect the processing component known as the Conductor in the rules-based scheduling system MEOW, has been extended with a remote option. This will enable MEOW to transmit jobs to a remote system, with the option of using slurm to orchestrate them. The new option is evaluated by comparing the execution time of using the remote solution without slurm, with that of existing components. Furthermore, the overhead of using various slurm methods for scheduling jobs with that of running the jobs remotely without using slurm is compared. It is shown that the remote solution adds a flat overhead to the execution time as both the number of jobs and size of the transmitted data increases, and that the overhead associated with using slurm on top of it is largely insignificant. This is considered to be an acceptable result given the test environment that was used.

show abstract