Orchestrating Docker Containers in the HPC Environment

Higgins, Joshua; Holmes, Violeta; Venters, Colin C.

doi:10.1007/978-3-319-20119-1_36

Cited by 39 publications

(15 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While Popper is tuned towards scientific publications, this also includes all experiments that have been performed for such a publication and can, therefore, be easily adapted for general experiments. Experiments may either be deployed locally with Docker [61] or in the cloud with Ansible [104].…”

Section: Recomputation and Reproducibilitymentioning

confidence: 99%

State of the Art and Future Trends in Data Reduction for High-Performance Computing

2020

JSFI

View full text Add to dashboard Cite

Research into data reduction techniques has gained popularity in recent years as storage capacity and performance become a growing concern. This survey paper provides an overview of leveraging points found in high-performance computing (HPC) systems and suitable mechanisms to reduce data volumes. We present the underlying theories and their application throughout the HPC stack and also discuss related hardware acceleration and reduction approaches. After introducing relevant use-cases, an overview of modern lossless and lossy compression algorithms and their respective usage at the application and file system layer is given. In anticipation of their increasing relevance for adaptive and in situ approaches, dimensionality reduction techniques are summarized with a focus on non-linear feature extraction. Adaptive approaches and in situ compression algorithms and frameworks follow. The key stages and new opportunities to deduplication are covered next. An unconventional but promising method is recomputation, which is proposed at last. We conclude the survey with an outlook on future developments.

show abstract

Section: Recomputation and Reproducibilitymentioning

confidence: 99%

State of the Art and Future Trends in Data Reduction for High-Performance Computing

2020

JSFI

View full text Add to dashboard Cite

show abstract

“…The tenant's hosts of our solution are containers. We opted for this operating system virtualization technology as it provides functionality similar to a VM but with a lighter footprint [9]. Each container (i.e., each tenant's host) has its own namespace (IP and MAC addresses, name, etc.)…”

Section: Virtualization Runtime: Achieving Isolationmentioning

confidence: 99%

Secure and Dependable Multi-Cloud Network Virtualization

Alaluna

Vial

Neves

et al. 2017

Proceedings of the 1st International Workshop on Security and Dependability of Multi-Domain Infrastructures

View full text Add to dashboard Cite

Existing multi-tenant network virtualization platforms have so far focused on the offer of conventional networking services by a single cloud provider. As such, they face limitations in terms of security and dependability, both in terms of the infrastructure itself and of the services offered to its customers. To address these challenges we present the design and implementation of Sirius, a network virtualization platform for multi-cloud environments. Contrary to existing solutions, Sirius considers not only connectivity and performance, but also security and dependability as first class citizens, leveraging from a substrate infrastructure composed of both public clouds and private data centers.

show abstract

“…These solutions most often take the form of HPC workloads containerized with Docker, allowing for the jobs to run in an arbitrary software environment that would not have been possible on the host system [23]. Security concerns with the use of Docker on a shared resource [19] have led to the rise of several containerization solutions built specifically for the needs of HPC, such as Singularity [24], Shifter [18], and CharlieCloud [25].…”

Section: Related Workmentioning

confidence: 99%

Continuous Integration and Delivery for HPC

Sampedro

Holt

Hauser

2018

Proceedings of the Practice and Experience on Advanced Research Computing

View full text Add to dashboard Cite

Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer [17] to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.

show abstract

Orchestrating Docker Containers in the HPC Environment

Cited by 39 publications

References 4 publications

State of the Art and Future Trends in Data Reduction for High-Performance Computing

State of the Art and Future Trends in Data Reduction for High-Performance Computing

Secure and Dependable Multi-Cloud Network Virtualization

Continuous Integration and Delivery for HPC

Contact Info

Product

Resources

About