Approaches to Distributed Execution of Scientific Workflows in Kepler

Płóciennik, Marcin; Żok, Tomasz; Altıntaş, Ilkay; Wang, Jianwu; Crawl, Daniel; Abramson, David; Imbeaux, F.; Guillerminet, B.; López‐Caniego, M.; Plasencia, Isabel Campos; Pych, W.; Cieciela̧g, P.; Palak, Bartek; Owsiak, Michał; Frauel, Y.

doi:10.3233/fi-2013-947

Cited by 14 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the Pegasus framework allows users to represent workflows at an abstract level while it takes care of the execution systems particulars [11]. Similarly, the main goal of the Kepler MS is to support different execution scenarios where a user can develop and use its modules to manage different execution behaviors in different environments, including private computational resources [25]. The unique features of Kepler are that the underlying workflow engine handles the provenance, reproducibility aspects of the code, performs orchestration of data flow, and automates execution on heterogeneous computing resources [33].…”

Section: The Computational Workflowmentioning

confidence: 99%

Micro-Workflows Data Stream Processing Model for Industrial Internet of Things

2021

JSFI

View full text Add to dashboard Cite

The fog computing paradigm has become prominent in stream processing for IoT systems where cloud computing struggles from high latency challenges. It enables the deployment of computational resources between the edge and cloud layers and helps to resolve constraints, primarily due to the need to react in real-time to state changes, improve the locality of data storage, and overcome external communication channels' limitations. There is an urgent need for tools and platforms to model, implement, manage, and monitor complex fog computing workflows. Traditional scientific workflow management systems (SWMSs) provide modularity and flexibility to design, execute, and monitor complex computational workflows used in smart industry applications. However, they are mainly focused on batch execution of jobs consisting of tightly coupled tasks. Integrating data streams into SWMSs of IoT systems is challenging. We proposed a microworkflow model to redesign the monolith architecture of workflow systems into a set of smaller and independent workflows that support stream processing. Micro-workflow is an independent data stream processing service that can be deployed on different layers of the fog computing environment. To validate the feasibility and practicability of the micro-workflow refactoring, we provide intensive experimental analysis evaluating the interval between sensor messages, the time interval required to create a message, between sending sensor message and receiving the message in SWMS, including data serialization, network latency, etc. We show that the proposed decoupling support of the independence of implementation, execution, development, maintenance, and cross-platform deployment, where each micro-workflow becomes a standalone computational unit, is a suitable mechanism for IoT stream processing.

show abstract

Section: The Computational Workflowmentioning

confidence: 99%

Micro-Workflows Data Stream Processing Model for Industrial Internet of Things

2021

JSFI

View full text Add to dashboard Cite

show abstract

“…Both, its multi-platform nature and its capability to integrate codes written in different programming languages, make Kepler suited for scientific communities. It can successfully orchestrate complex workflows and perform parametric scans [7]. There are also Kepler modules which enable usage of grid and cloud middleware to submit remote jobs [8][9][10] using, e.g.…”

Section: Overviewmentioning

confidence: 99%

“…It includes many large codes including ASCOT [13] or BIT [14][15][16] and needs substantial provision of computing resources and the IMAS library to be executed. In the past, access to HPC resources was realised using the HPC2K tool (HPC to Kepler) [7]. In this approach, a simulation was conducted by the code executed on the login node, and the communication with computation nodes was performed using a UNICORE-based [11] queuing system.…”

Section: Integrated Modelling With Hpc Requirementsmentioning

confidence: 99%

Progress in the transferability of fusion workflows across HPC systems

Gutierrez-Milla¹,

Żok²,

Owsiak³

et al. 2021

Plasma Phys. Control. Fusion

View full text Add to dashboard Cite

Understanding the behaviour of high-temperature plasmas is one of the main pillars in the development of fusion energy. It involves the development, validation, and use of several numerical models to describe complex physical processes and their interactions. Integrated modelling brings different models together, coupling them via suitable interfaces. Often, there is a need to include specific physics phenomena, such as magnetohydrodynamics, plasma turbulence or transport. They are computationally demanding and require modern supercomputers for their analysis. In such cases, the complexity of running integrated modelling workflows, including the use of supercomputers should be managed transparently, hidden to the final user. In this paper, we present and implement a scheme to tackle the execution of large fusion workflows on modern supercomputers using container technologies and a tool for their remote submission. We successfully packed in a container image a very complex environment: international thermonuclear experimental reactor (ITER) integrated modelling & analysis suite (IMAS). Moreover, we run high-performance computing codes up to 3072 cores with performance loss of 3%. The ITER H&CD worklfow was executed on Marconi and for the first time ran in a cluster without an installation of the IMAS framework. The presented capabilities have demonstrated the feasibility of our approach on Marconi-Fusion, the European High-Performance Computer for fusion applications and have tested a relevant application within the integrated modelling community.

show abstract