Bridging the Gap Between HPC and Cloud Using HyperFlow and PaaSage

Hoppe, Dennis; Sandoval, Yosandra; Sulistio, Anthony; Malawski, Maciej; Baliś, Bartosz; Pawlik, Maciej; Figiela, Kamil; Król, Dariusz; Orzechowski, Michał; Kitowski, Jacek; Bubak, Marian

doi:10.1007/978-3-319-78024-5_38

Cited by 3 publications

(2 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Orzechowski et al [208] describe the deployment of HyperFlow [209], a SWFMS for distributed systems, on an automatically provisioned Kubernetes [147] cluster to manage and scale a container infrastructure for the workflow. Hoppe et al [210] couple Hyperflow with CAMEL [115] to foster the utilization of cloud and HPC resources within a single workflow.…”

Section: Infrastructure Aware Workflow Managementmentioning

confidence: 99%

“…Section 3.3 listed even further workflow approaches with special interest in using distributed computing resources [167,170,[193][194][195][196][197][198][199][200][201][202]. The related work shows approaches combining workflows and their execution with existing orchestration techniques [207,209] or even cloud modeling languages [159,[210][211][212]. Furthermore, approaches exist that utilize a reflective middleware for workflow models using annotations to manage deployed resources at runtime [214,215].…”

Section: Summary and Research Gapmentioning

confidence: 99%

See 1 more Smart Citation

Scientific Workflow Execution Using a Dynamic Runtime Model

Erbel¹

View full text Add to dashboard Cite

Research projects consist of several kinds of steps covering, e.g., individual procedures to gather data, as well as different ways to process and analyze it. Moreover, the individual steps of the project consist of a sequence of tasks that together form a workflow. With the continuous advancements in computer science, scientists have more and more access to different kinds of infrastructures and tools which are suited for different types of experiments. Cloud computing is one of the premier infrastructures to perform experiments on, as it provides flexible, on-demand computing resources that are off-premise. Still, a uniform and platform-independent orchestration of these resources remains challenging, especially when various infrastructures and human interactions are required throughout the execution of a scientific workflow.This thesis provides an approach to allow scientists to define infrastructural resources for individual tasks within their workflows and dynamically shift them throughout the workflow execution. To reach this goal, we couple recent advancements in cloud orchestration with runtime models and open cloud standards. Combined, we aim for highly tailored workflows while fostering the reuse of already existing methodologies built around the standards. To realize this objective, we build a cloud runtime model orchestration process based on the Open Cloud Computing Interface (OCCI) standard. We extend the OCCI data model with workflow elements and corresponding capabilities to model cloud deployments for individual workflow tasks. This allows forming a runtime workflow model that can be coupled to different systems such as production clouds or simulation environments. To demonstrate the feasibility of the approach, we perform several experiments to validate the standard conform orchestration process and assess the applicability of the runtime workflow model coupled to cloud infrastructures.Our studies show that runtime models are a suitable knowledge base for adaptive behavior including the modeling and runtime representation of highly tailored workflows. We observe that the orchestration of cloud deployments and the execution of workflows follow a reoccurring pattern which can be described via a sequence of runtime states. Especially the uniform interface of OCCI allows for an automatic management and reuse of existing systems and standards. By monitoring and reflecting operational properties, the runtime model fosters decision-making processes not only for self-adaptive systems but also for human users. We show that the runtime model enables human-in-the-loop activities, allowing, e.g., to influence the control flow or the parallelization of workflow tasks at runtime. Furthermore, the runtime model can be attached to different environments allowing to test adaptation and workflow behavior. I especially want to thank Prof. Dr. Jens Grabowski who took me into his group and allowed me to study the wonders of distributed computing. He always provided me with feedback on how to improve and guided me th...

show abstract

Section: Infrastructure Aware Workflow Managementmentioning

confidence: 99%

Section: Summary and Research Gapmentioning

confidence: 99%

Scientific Workflow Execution Using a Dynamic Runtime Model

Erbel¹

View full text Add to dashboard Cite

show abstract

Scientific workflow execution in the cloud using a dynamic runtime model

Erbel

Grabowski

2023

Softw Syst Model

View full text Add to dashboard Cite

To explain specific phenomena, scientists perform a sequence of tasks, e.g., to gather, analyze and interpret data, forming a scientific workflow. Depending on the complexity of the workflow, scientists require access to various kinds of tools, applications and infrastructures for individual tasks. Current approaches are often limited to managing these resources at design time, requiring the scientist to preemptively set up applications essential for their workflow. Therefore, a dynamic provisioning and configuration of computing resources are required that fulfills these needs at runtime. In this paper, we present a dynamic runtime model that couples workflow tasks with their individual applications and infrastructure requirements. This runtime model is used as a knowledge base by a model-driven workflow execution engine orchestrating the sequence of tasks and their infrastructure. We exhibit that the simplicity of the runtime model supports the creation of highly tailored infrastructures, the integration of self-developed applications, as well as a human-in-the-loop allowing scientists to monitor and interact with the workflow at runtime. To tackle the heterogeneity of cloud provider interfaces, we implement the workflow runtime model by extending the Open Cloud Computing Interface cloud standard, which provides an extensible data model as well as a uniform interface to manage cloud resources. We demonstrate the applicability of our approach using three case studies and discuss the benefits of the runtime model from a user and system perspective.

show abstract

Developing AI Applications for the HPC-Cloud Continuum with ColonyOS

Kristiansson,

Wikfeldt

2024

2024 23rd International Symposium on Parallel and Distributed Computing (ISPDC)

View full text Add to dashboard Cite

Bridging the Gap Between HPC and Cloud Using HyperFlow and PaaSage

Cited by 3 publications

References 8 publications

Scientific Workflow Execution Using a Dynamic Runtime Model

Scientific Workflow Execution Using a Dynamic Runtime Model

Scientific workflow execution in the cloud using a dynamic runtime model

Developing AI Applications for the HPC-Cloud Continuum with ColonyOS

Contact Info

Product

Resources

About