Seamlessly Managing HPC Workloads Through Kubernetes

López-Huguet, Sergio; Segrelles, J. Damià; Kasztelnik, Marek; Bubak, Marian; Blanquer, Ignácio

doi:10.1007/978-3-030-59851-8_20

Cited by 12 publications

(3 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Overseeing HPC responsibilities through Kubernets Ignacio Bubak and Blanquer 21 have been developed to integrate the managing jobs on HPC infrastructures. An open‐source tool known as the HPC connector can seamlessly manage workloads by coordinating the full‐time cycle of jobs with the HPC system's workload manager.…”

Section: Literature Surveymentioning

confidence: 99%

Containerized deep learning in agriculture: Orchestrating GoogleNet with Kubernetes on high performance computing

Hasan,

Khan

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

SummarySmart Farming has become a cornerstone of modern agriculture, offering data‐driven insights and automation that optimize resource utilization and increase crop yields. The use of cutting‐edge technologies in agriculture has given rise to Smart Farming, which has transformed traditional farming practices into efficient, data‐driven operations. This paper explores the synergy between high‐performance computing (HPC) systems, Kubernetes orchestration, GoogleNet architecture, and containerization to redefine the future of farming. At the heart of this transformation lies the GoogleNet architecture, a deep learning powerhouse recognized for its efficiency and accuracy in image recognition tasks. The orchestration capabilities of Kubernetes, a versatile tool for managing containerized workloads efficiently on HPC clusters. Hence, in this work, we investigate the intricacies of deploying GoogleNet‐based deep learning models within containerized environments orchestrated by Kubernetes on HPC infrastructure. It explores resource optimization, scalability, security, and adaptability, all tailored to the unique demands of the agricultural domain to evaluate the effectiveness of the given technique it is compared with the existing techniques namely Hermes, Horus, CYBELE, and RZ‐SHAN. The attained ranges of proposed method of various measures of accuracy, precision, recall, and F1‐score are 98.65%, 97.45%, 97.87%, and 98.12% for the Pilot Wheat Ear dataset. Also, the processing time for the proposed approach is 181.50 and 120.2 m for the Pilot Wheat Ear Dataset and the Pilot soya bean farming dataset. The latency of the proposed approach attains a lower value of 1.5 and 1.1 s pilot soya bean farming dataset and Pilot Wheat Ear dataset. The experimental outcome demonstrates the efficiency of the proposed approaches to improve Smart Farming agriculture.

show abstract

Section: Literature Surveymentioning

confidence: 99%

Containerized deep learning in agriculture: Orchestrating GoogleNet with Kubernetes on high performance computing

Hasan,

Khan

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…In [20], the authors use a utility called hpc-connector that acts as an HPC job proxy: Users submit Kubernetes jobs with specific settings, and the hpcconnector forwards them to the HPC cluster, monitors their execution, and collects their results. Similarly, in [23], a Kubernetes installation is interfaced to a Torque-based HPC cluster, using a custom tool called Torque-Operator.…”

Section: Related Workmentioning

confidence: 99%

Running Kubernetes Workloads on HPC

Chazapis,

Nikolaidis,

Marazakis

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Cloud and HPC increasingly converge in hardware platform capabilities and specifications, nevertheless still largely differ in the software stack and how it manages available resources. The HPC world typically favors Slurm for job scheduling, whereas Cloud deployments rely on Kubernetes to orchestrate container instances across nodes. Running hybrid workloads is possible by using bridging mechanisms that submit jobs from one environment to the other. However, such solutions require costly data movements, while operating within the constraints set by each setup's network and access policies. In this work, we explore a design that enables running unmodified Kubernetes workloads directly on HPC. With High-Performance Kubernetes (HPK), users deploy their own private Kubernetes "mini Clouds", which internally convert container lifecycle management commands to use the system-level Slurm installation for scheduling and Singularity/Apptainer as the container runtime. We consider this approach to be practical for deployment in HPC centers, as it requires minimal pre-configuration and retains existing resource management and accounting policies. HPK provides users with an effective way to utilize resources by a combination of well-known tools, APIs, and more interactive and user-friendly interfaces as is common practice in the Cloud domain, as well as seamlessly combine Cloud-native tools with HPC jobs in converged, containerized workflows.

show abstract

“…In this work, the authors combine a Kubernetes cluster hosted in Spain with the supercomputer Prometheus [34] hosted in Poland, which uses the job scheduler Slurm. This work [103] was published in the conference ISC High Performance, which is rated as CORE C and has not defined a class yet in GII-GRIN-SCIE conference rating.…”

Section: Organization Of This Documentmentioning

confidence: 99%

Elastic, Interoperable and Container-based Cloud Infrastructures for High Performance Computing

Huguet¹

View full text Add to dashboard Cite

show abstract

Seamlessly Managing HPC Workloads Through Kubernetes

Cited by 12 publications

References 15 publications

Containerized deep learning in agriculture: Orchestrating GoogleNet with Kubernetes on high performance computing

Containerized deep learning in agriculture: Orchestrating GoogleNet with Kubernetes on high performance computing

Running Kubernetes Workloads on HPC

Elastic, Interoperable and Container-based Cloud Infrastructures for High Performance Computing

Contact Info

Product

Resources

About