Data-Intensive HPC Tasks Scheduling with SDN to Enable HPC-as-a-Service

Jamalian, Saba; Rajaei, Hassan

doi:10.1109/cloud.2015.85

Cited by 11 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Software Defined Networks (SDN) [12]- [14] is a new technology to manage the Virtual Networks through Software controller as a replacement for the physical routers controls. SDN separates the data plane from the controller plane of the network to allow much flexible and dynamic network management in the virtual Networks.…”

Section: Related Workmentioning

confidence: 99%

Deployment and Analysis of a Hybrid Shared/Distributed-Memory Parallel Visualization Tool for 3-D Oil Reservoir Grid on OpenStack Cloud Computing

et al. 2020

View full text Add to dashboard Cite

The main goal of oil reservoir management is to provide more efficient, price-effective and environmentally more secure oil production. Oil production management includes an accurate characterization of the reservoir and strategies that involve interactions between reservoir data and human assessment. Hence, it is important to graphically visualize and handle massive data sets of oil and gas pressure / saturation levels to help decision makers in statistical analysis, history matching and recovery of hydrocarbons of the reservoir. In this article, we experimentally study the parallelization of intensive computation for a 3-D (three dimensional) oil reservoir data visualization tool. For this tool, we develop and implement a transformation and lighting model to visualize and react with the grid. Herein, we propose a hybrid (shared memory and distributed memory) parallelization technique to adapt with the data processing scalability. We tested these implementations on OpenStack Cloud Virtual Cluster. Our results indicate that although the virtual platform adds overhead for running parallel implementations, utilizing knowledge of the VM location on the compute host and network traffic among VMs to deploy the virtual environment can achieve significant performance enhancements. Hybrid parallel implementation using large data size can achieve 70× speedup over serial execution without owning a costly HPC infrastructure as the conventional parallel processing deployment model.INDEX TERMS Cloud computing, data visualization tool, HPC, hybrid (distributed/shared)-memoryparallel programming, MPI, multi-threading, OpenStack.

show abstract

Section: Related Workmentioning

confidence: 99%

Deployment and Analysis of a Hybrid Shared/Distributed-Memory Parallel Visualization Tool for 3-D Oil Reservoir Grid on OpenStack Cloud Computing

et al. 2020

View full text Add to dashboard Cite

show abstract

“…There are examples of cloud platforms in the literature that use the previous architecture scheme. In [14], the authors present an architecture to process Internet of Things (IoT) data collected from smart agriculture. In our previous works, we presented a cloud architecture for data analysis [19] and for processing medical imaging [18].…”

Section: Architecturementioning

confidence: 99%

“…The HPC infrastructure is the Prometheus supercomputer [10], which is located in the 174 th position of the TOP500 list (June 2019). Prometheus cluster provides REST API to interact with SLURM called Rimrock (Robust Remote Process Controller) 13 and PLGData service 14 to interact with the file system.…”

Section: Use Case: Segmentation Of Neuroblastoma Tumoursmentioning

confidence: 99%

Seamlessly Managing HPC Workloads Through Kubernetes

López-Huguet

Segrelles

Kasztelnik

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper describes an approach to integrate the jobs management of High Performance Computing (HPC) infrastructures in cloud architectures by managing HPC workloads seamlessly from the cloud job scheduler. The paper presents hpc-connector, an open source tool that is designed for managing the full life cycle of jobs in the HPC infrastructure from the cloud job scheduler interacting with the workload manager of the HPC system. The key point is that, thanks to running hpc-connector in the cloud infrastructure, it is possible to reflect in the cloud infrastructure, the execution of a job running in the HPC infrastructure managed by hpc-connector. If the user cancels the cloud-job, as hpc-connector catches Operating System (OS) signals (for example, SIGINT), it will cancel the job in the HPC infrastructure too. Furthermore, it can retrieve logs if requested. Therefore, by using hpc-connector, the cloud job scheduler can manage the jobs in the HPC infrastructure without requiring any special privilege, as it does not need changes on the Job scheduler. Finally, we perform an experiment training a neural network for automated segmentation of Neuroblastoma tumours in the Prometheus supercomputer using hpc-connector as a batch job from a Kubernetes infrastructure.

show abstract

“…For parallel programming purpose, the Message-Passing Interface 19,41 (MPI) library and OpenMP can be used. As a warm-up exercise, we can assign students with matrix multiplication benchmark.…”

Section: Hpc Educational Examplesmentioning

confidence: 99%

HPC as a Service in Education

Rajaei¹,

Jamalian²

2016 ASEE Annual Conference &Amp; Exposition Proceedings

Self Cite

View full text Add to dashboard Cite

Advances in Cloud Computing have opened many chapters in Information Technology. Numerous service platforms offer clients of the cloud ease of use and flexibility of using the provided services. Education with billions of potential users worldwide is a major target. An emerging service called HPC-as-a-Service (HPCaaS) targets Science, Technology, Engineering, and Math (STEM) users. In this paper we discuss the use of HPCaaS platform in STEM education. We argue that such a service can significantly alleviate a major obstacle in teaching parallel programming for the STEM students. Cloud computing provides unique benefits such as resource pooling, cost efficiency, availability, and large computational power. These features have attracted scientists, engineers, scholars, and the High Performance Computing (HPC) users like a magnet towards the Cloud. However, HPC programs often consume large number of collaborating processors to reduce the execution time, where synchronization between these processors and the communication overhead among them can become real challenge even on dedicated and special hardware, but worse on shared and virtualized platform like cloud. As a result, moving HPC applications to the cloud can adversely impact the abovenamed difficulties with additional issues, primarily virtualization, multitenancy, and network latency. One solution is a new cloud service known as HPC-as-a-Service. In this paper, we present an HPCaaS platform called ASETS which uses Software Defined Networking (SDN) technologies to smooth the execution of parallel tasks in the cloud. Further, we provide application examples that could be used in a typical introductory parallel programing course. We argue that HPCaaS platform like ASETS can significantly benefit the users of HPC in the cloud as if their program is running on a dedicated hardware in their own laboratory. This is especially advantageous for students and educators who need not to deal with the underlying complexities of the cloud.

show abstract

Data-Intensive HPC Tasks Scheduling with SDN to Enable HPC-as-a-Service

Cited by 11 publications

References 21 publications

Deployment and Analysis of a Hybrid Shared/Distributed-Memory Parallel Visualization Tool for 3-D Oil Reservoir Grid on OpenStack Cloud Computing

Deployment and Analysis of a Hybrid Shared/Distributed-Memory Parallel Visualization Tool for 3-D Oil Reservoir Grid on OpenStack Cloud Computing

Seamlessly Managing HPC Workloads Through Kubernetes

HPC as a Service in Education

Contact Info

Product

Resources

About