Deploying Scientific Al Networks at Petaflop Scale on Secure Large Scale HPC Production Systems with Containers

Bustos, David; Vallecorsa, S.

doi:10.1145/3394277.3401850

Cited by 8 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Over the next decade, scientists will see a 10-100 times increase in sensitivity and resolution from their instruments, necessitating a comparable scale-up in data storage and processing capacity. The data derived by these upgraded instruments will push Moore's law to its limits, posing a threat to conventional operating models predicated primarily on HPC in data centers [167,168]. Conventional HPC architectures were developed for simulation-based methods like computational fluid dynamics.…”

Section: High-performance Computingmentioning

confidence: 99%

Intelligent Computing: The Latest Advances, Challenges, and Future

Zhu

et al. 2023

Intell Comput

View full text Add to dashboard Cite

Computing is a critical driving force in the development of human civilization. In recent years, we have witnessed the emergence of intelligent computing, a new computing paradigm that is reshaping traditional computing and promoting digital revolution in the era of big data, artificial intelligence, and internet of things with new computing theories, architectures, methods, systems, and applications. Intelligent computing has greatly broadened the scope of computing, extending it from traditional computing on data to increasingly diverse computing paradigms such as perceptual intelligence, cognitive intelligence, autonomous intelligence, and human–computer fusion intelligence. Intelligence and computing have undergone paths of different evolution and development for a long time but have become increasingly intertwined in recent years: Intelligent computing is not only intelligence oriented but also intelligence driven. Such cross-fertilization has prompted the emergence and rapid advancement of intelligent computing. Intelligent computing is still in its infancy, and an abundance of innovations in the theories, systems, and applications of intelligent computing is expected to occur soon. We present the first comprehensive survey of literature on intelligent computing, covering its theory fundamentals, the technological fusion of intelligence and computing, important applications, challenges, and future perspectives. We believe that this survey is highly timely and will provide a comprehensive reference and cast valuable insights into intelligent computing for academic and industrial researchers and practitioners.

show abstract

Section: High-performance Computingmentioning

confidence: 99%

Intelligent Computing: The Latest Advances, Challenges, and Future

Zhu

et al. 2023

Intell Comput

View full text Add to dashboard Cite

show abstract

“…In [16] authors focus on applications oriented to deep learning, for which they use Charliecloud [12]. Subsequently, a similar workflow is applied to the training of a deep learning model in the field of particle physics [17]. The developed workflow allows them to take advantage of the hundreds of CPUs available in their cluster.…”

Section: Related Workmentioning

confidence: 99%

“…This experiment was executed in a different cluster from the previous one, however it was not necessary to apply substantial changes to the workflow. The image built for this experiment 17 was developed using as basis the image from the TensorFlow Benchmark experiment, solely adding the software required for this second experiment. The scheduling of the jobs has been done in the same way in both experiments (following the indications of Sect.…”

Section: Empirical Statistical Downscalingmentioning

confidence: 99%

“…Sometimes the code needs to be adapted and special versions of software may need to be installed in order to take advantage of specialized hardware. For this reason, several workflows have recently been developed [15][16][17] to combine in a single flow the entire process required to run an experiment, from the development of the experiment to its deployment on a cluster, using the previously mentioned containerization tools.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A container-based workflow for distributed training of deep learning algorithms in HPC clusters

2022

View full text Add to dashboard Cite

Deep learning has been postulated as a solution for numerous problems in different branches of science. Given the resource-intensive nature of these models, they often need to be executed on specialized hardware such graphical processing units (GPUs) in a distributed manner. In the academic field, researchers get access to this kind of resources through High Performance Computing (HPC) clusters. This kind of infrastructures make the training of these models difficult due to their multi-user nature and limited user permission. In addition, different HPC clusters may possess different peculiarities that can entangle the research cycle (e.g., libraries dependencies). In this paper we develop a workflow and methodology for the distributed training of deep learning models in HPC clusters which provides researchers with a series of novel advantages. It relies on udocker as containerization tool and on Horovod as library for the distribution of the models across multiple GPUs. udocker does not need any special permission, allowing researchers to run the entire workflow without relying on any administrator. Horovod ensures the efficient distribution of the training independently of the deep learning framework used. Additionally, due to containerization and specific features of the workflow, it provides researchers with a cluster-agnostic way of running their models. The experiments carried out show that the workflow offers good scalability in the distributed training of the models and that it easily adapts to different clusters.

show abstract

“…Their results indicate that running computationally intensive jobs on CPUs/GPUs has little overhead compared to running the jobs directly on top of the operating system. David Brayford et al have successfully used containers to deploy three-dimensional convolutional GAN (3DGAN) with petaflop performance on High-Performance Computing (HPC) resources[24]. They state that using containers allowed them to use HPC clusters without sacrificing the security of the cluster, and helped the execution…”

mentioning

confidence: 99%

Containerization of Model Fitting Workloads over Spatial Datasets

Warushavithana

Mitra

Arabi

et al. 2021

2021 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

CONTAINERIZATION OF MODEL FITTING WORKLOADS OVER SPATIAL DATASETSSpatial data volumes have grown exponentially over the past several years. The number of domains in which spatial data are extensively leveraged include atmospheric sciences, environmental monitoring, ecological modeling, epidemiology, sociology, commerce, and social media among others. These data are often used to understand phenomena and inform decision making by fitting models to them. In this study, we present our methodology to fit models at scale over spatial data.Our methodology encompasses segmentation, spatial similarity based on the dataset(s) under consideration, and transfer learning schemes that are informed by the spatial similarity to train models faster while utilizing fewer resources. We consider several model fitting algorithms and execution within containerized environments as we profile the suitability of our methodology. Our benchmarks validate the suitability of our methodology to facilitate faster, resource-efficient training of models over spatial data. Chapter 5Conclusions and Future Work .

show abstract

Deploying Scientific Al Networks at Petaflop Scale on Secure Large Scale HPC Production Systems with Containers

Cited by 8 publications

References 15 publications

Intelligent Computing: The Latest Advances, Challenges, and Future

Intelligent Computing: The Latest Advances, Challenges, and Future

A container-based workflow for distributed training of deep learning algorithms in HPC clusters

Containerization of Model Fitting Workloads over Spatial Datasets

Contact Info

Product

Resources

About