HPC Container Runtimes have Minimal or No Performance Impact

Torrez, Alfred; Randles, Timothy; Priedhorsky, Reid

doi:10.1109/canopie-hpc49598.2019.00010

Cited by 39 publications

(15 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A container runtime provides an isolated file system using the container image to the container instance as if it is dedicated to the host. By using resource isolation features, e.g., namespace in Linux kernel, rather than hardware emulation, e.g., virtual machines, a container runtime can execute a container instance as efficiently as the host process [60]. In the bioinformatics field, Docker engine and Singularity are widely used for data analysis applications.…”

Section: Containersmentioning

confidence: 99%

Practical guide for managing large-scale human genome data in research

et al. 2020

View full text Add to dashboard Cite

Studies in human genetics deal with a plethora of human genome sequencing data that are generated from specimens as well as available on public domains. With the development of various bioinformatics applications, maintaining the productivity of research, managing human genome data, and analyzing downstream data is essential. This review aims to guide struggling researchers to process and analyze these large-scale genomic data to extract relevant information for improved downstream analyses. Here, we discuss worldwide human genome projects that could be integrated into any data for improved analysis. Obtaining human whole-genome sequencing data from both data stores and processes is costly; therefore, we focus on the development of data format and software that manipulate whole-genome sequencing. Once the sequencing is complete and its format and data processing tools are selected, a computational platform is required. For the platform, we describe a multi-cloud strategy that balances between cost, performance, and customizability. A good quality published research relies on data reproducibility to ensure quality results, reusability for applications to other datasets, as well as scalability for the future increase of datasets. To solve these, we describe several key technologies developed in computer science, including workflow engine. We also discuss the ethical guidelines inevitable for human genomic data analysis that differ from model organisms. Finally, the future ideal perspective of data processing and analysis is summarized.

show abstract

Section: Containersmentioning

confidence: 99%

Practical guide for managing large-scale human genome data in research

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Memory overhead measurements were negligible when running the workloads through the Charliecloud runtime environment compared to running directly on the host system as illustrated in table 9. A.Torrez et al [32] describe the performance impacts of HPC container runtimes on HPC benchmarks such as High Performance LINPACK (HPL), High Performance Conjugate Gradient (HPCG) and STREAM.…”

Section: Overheadsmentioning

confidence: 99%

Deploying Scientific Al Networks at Petaflop Scale on Secure Large Scale HPC Production Systems with Containers

Bustos

Vallecorsa

2020

Proceedings of the Platform for Advanced Scientific Computing Conference

View full text Add to dashboard Cite

There is an ever-increasing need for computational power to train complex artificial intelligence (AI) & machine learning (ML) models to tackle large scientific problems. High performance computing (HPC) resources are required to efficiently compute and scale complex models across tens of thousands of compute nodes. In this paper, we discuss the issues associated with the deployment of machine learning frameworks on large scale secure HPC systems and how we successfully deployed a standard machine learning framework on a secure large scale HPC production system, to train a complex three-dimensional convolutional GAN (3DGAN), with petaflop performance. 3DGAN is an example from the high energy physics domain, designed to simulate the energy pattern produced by showers of secondary particles inside a particle detector on various HPC systems.

show abstract

“…Containers are an increasingly popular way to meet these needs. 1 While containers have been deployed across a wide array of supercomputers in recent years [6,17,33,49,55], containerized workloads still represent a distinct minority of supercomputer cycles [4]. We believe a key reason is the difficulty of building container images with unprivileged tools.…”

Section: Introductionmentioning

confidence: 99%

Minimizing privilege for building HPC containers

Priedhorsky,

Canon,

Randles

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

HPC centers face increasing demand for software flexibility, and there is growing consensus that Linux containers are a promising solution. However, existing container build solutions require root privileges and cannot be built directly on HPC resources. This limitation is compounded as supercomputer diversity expands and HPC architectures become more dissimilar from commodity computing resources. Our evaluation of available options suggests this problem can best be solved with low-privilege containers. We detail Linux kernel features for varying container privilege and compare two open-source implementations, mostly-unprivileged rootless Podman and fully-unprivileged Charliecloud. Our analysis demonstrates that low-privilege container build on HPC resources works now and will continue to improve, giving normal users a better workflow to securely and correctly build containers. Minimizing privilege in this way can improve HPC user and developer productivity as well as reduce support workload for exascale applications.

show abstract

HPC Container Runtimes have Minimal or No Performance Impact

Cited by 39 publications

References 12 publications

Practical guide for managing large-scale human genome data in research

Practical guide for managing large-scale human genome data in research

Deploying Scientific Al Networks at Petaflop Scale on Secure Large Scale HPC Production Systems with Containers

Minimizing privilege for building HPC containers

Contact Info

Product

Resources

About