Scientific workflow orchestration interoperating HTC and HPC resources

Cabellos, Luis; Campos, Isabel; Fernández-del-Castillo, Enol; Owsiak, Michał; Palak, Bartek; Płóciennik, Marcin

doi:10.1016/j.cpc.2010.12.020

Cited by 12 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such workflows imply the usage of both gLite and UNICORE modules in one workflow. Examples of such workflows built have been demonstrated within the EUFORIA project [27].…”

Section: B Unicore Modulementioning

confidence: 98%

Workflows orchestration in distributed computing infrastructures

Płóciennik

Żok

Gómez-Iglesias

et al. 2012

2012 International Conference on High Performance Computing &Amp; Simulation (HPCS)

View full text Add to dashboard Cite

This paper presents an overview of the new developments carried out to offer a reliable and efficient support for different computing infrastructures in the Kepler workflow orchestration system. The aim of the work is to help scientists to transparently use these infrastructures regardless of the underlying middleware. We introduce new complex workflow scenarios developed in the context of the EU EGI Inspire project and their exploitation by two nuclear fusion applications. The presented use cases represent typical scenarios which are common also in other fields and could be easily reused for other applications.

show abstract

“…Such workflows imply the usage of both gLite and UNICORE modules in one workflow. Examples of such workflows built have been demonstrated within the EUFORIA project [27].…”

Section: B Unicore Modulementioning

confidence: 98%

Workflows orchestration in distributed computing infrastructures

Płóciennik

Żok

Gómez-Iglesias

et al. 2012

2012 International Conference on High Performance Computing &Amp; Simulation (HPCS)

View full text Add to dashboard Cite

show abstract

“…The Serpens suite [8] contains two modules: gLite and UNICORE modules. These module act as adapters to the corresponding grid middleware, adding additional fault tolerance mechanism on top.…”

Section: Serpensmentioning

confidence: 99%

Approaches to Distributed Execution of Scientific Workflows in Kepler

Płóciennik

Żok

Altıntaş

et al. 2013

Fundamenta Informaticae

Self Cite

View full text Add to dashboard Cite

The Kepler scientific workflow system enables creation, execution and sharing of workflows across a broad range of scientific and engineering disciplines while also facilitating remote and distributed execution of workflows. In this paper, we present and compare different approaches to distributed execution of workflows using the Kepler environment, including a distributed dataparallel framework using Hadoop and Stratosphere, and Cloud and Grid execution using Serpens, Nimrod/K and Globus actors. We also present real-life applications in computational chemistry, bioinformatics and computational physics to demonstrate the usage of different distributed computing capabilities of Kepler in executable workflows. We further analyze the differences of each approach and provide a guidance for their applications.

show abstract

“…Most scientific workloads combine requirements that could be efficiently addressed using a combination of High-Throughput Computing (HTC) and High-Performance Computing (HPC) workloads [7][8] [20]. Focusing on Medical Imaging, HPC is extensively used for artificial intelligence model building and simulation.…”

Section: Introductionmentioning

confidence: 99%

Seamlessly Managing HPC Workloads Through Kubernetes

López-Huguet

Segrelles

Kasztelnik

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper describes an approach to integrate the jobs management of High Performance Computing (HPC) infrastructures in cloud architectures by managing HPC workloads seamlessly from the cloud job scheduler. The paper presents hpc-connector, an open source tool that is designed for managing the full life cycle of jobs in the HPC infrastructure from the cloud job scheduler interacting with the workload manager of the HPC system. The key point is that, thanks to running hpc-connector in the cloud infrastructure, it is possible to reflect in the cloud infrastructure, the execution of a job running in the HPC infrastructure managed by hpc-connector. If the user cancels the cloud-job, as hpc-connector catches Operating System (OS) signals (for example, SIGINT), it will cancel the job in the HPC infrastructure too. Furthermore, it can retrieve logs if requested. Therefore, by using hpc-connector, the cloud job scheduler can manage the jobs in the HPC infrastructure without requiring any special privilege, as it does not need changes on the Job scheduler. Finally, we perform an experiment training a neural network for automated segmentation of Neuroblastoma tumours in the Prometheus supercomputer using hpc-connector as a batch job from a Kubernetes infrastructure.

show abstract

Scientific workflow orchestration interoperating HTC and HPC resources

Cited by 12 publications

References 15 publications

Workflows orchestration in distributed computing infrastructures

Workflows orchestration in distributed computing infrastructures

Approaches to Distributed Execution of Scientific Workflows in Kepler

Seamlessly Managing HPC Workloads Through Kubernetes

Contact Info

Product

Resources

About