Performance Analysis of Applications using Singularity Container on SDSC Comet

Le, Emily; Paz, David

doi:10.1145/3093338.3106737

Cited by 26 publications

(9 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An assessment of natural methane emissions simulated by CTEM is presented in who use a one-box model of atmospheric methane together with prescribed anthropogenic and geological sources to reproduce atmospheric methane concentrations consistent with the observational record over the historical period. CLASS-CTEM has also contributed regularly to the Global Carbon Project's annual carbon budget analyses since 2016 (Le Quéré et al, 2016, 2018b.…”

Section: Model Biogeochemistry: Ctemmentioning

confidence: 99%

“…For CLASSIC, we use a Singularity container (https:// www.sylabs.io/, last access: 19 November 2019), which is portable to Linux, Mac, and Windows systems as well as HPC clusters, for which Singularity was specifically designed (Kurtzer et al, 2017). Tests of Singularity's performance on HPC benchmarks have demonstrated a negligible performance overhead (Le and Paz, 2017). Singularity is already installed on many national, university and government HPC centres including Compute Canada, Argonne National Labs, Fermilab, National Supercomputer Centre (Sweden), amongst others.…”

Section: Containerizationmentioning

confidence: 99%

See 1 more Smart Citation

CLASSIC v1.0: the open-source community successor to the Canadian Land Surface Scheme (CLASS) and the Canadian Terrestrial Ecosystem Model (CTEM) – Part 1: Model framework and site-level performance

et al. 2020

View full text Add to dashboard Cite

Abstract. Recent reports by the Global Carbon Project highlight large uncertainties around land surface processes such as land use change, strength of CO2 fertilization, nutrient limitation and supply, and response to variability in climate. Process-based land surface models are well suited to address these complex and emerging global change problems but will require extensive development and evaluation. The coupled Canadian Land Surface Scheme and Canadian Terrestrial Ecosystem Model (CLASS-CTEM) framework has been under continuous development by Environment and Climate Change Canada since 1987. As the open-source model of code development has revolutionized the software industry, scientific software is experiencing a similar evolution. Given the scale of the challenge facing land surface modellers, and the benefits of open-source, or community model, development, we have transitioned CLASS-CTEM from an internally developed model to an open-source community model, which we call the Canadian Land Surface Scheme including Biogeochemical Cycles (CLASSIC) v.1.0. CLASSIC contains many technical features specifically designed to encourage community use including software containerization for serial and parallel simulations, extensive benchmarking software and data (Automated Model Benchmarking; AMBER), self-documenting code, community standard formats for model inputs and outputs, amongst others. Here, we evaluate and benchmark CLASSIC against 31 FLUXNET sites where the model has been tailored to the site-level conditions and driven with observed meteorology. Future versions of CLASSIC will be developed using AMBER and these initial benchmark results to evaluate model performance over time. CLASSIC remains under active development and the code, site-level benchmarking data, software container, and AMBER are freely available for community use.

show abstract

Section: Model Biogeochemistry: Ctemmentioning

confidence: 99%

Section: Containerizationmentioning

confidence: 99%

CLASSIC v1.0: the open-source community successor to the Canadian Land Surface Scheme (CLASS) and the Canadian Terrestrial Ecosystem Model (CTEM) – Part 1: Model framework and site-level performance

et al. 2020

View full text Add to dashboard Cite

show abstract

“…While containerization started with cloud computing [2,8] and not with HPC, scientific workloads have started adopting containers to manage complex toolchains and software dependencies to ease deployment in multiple Linux variants [15]. Widely deployed container systems that have found use in HPC include Docker [5,9,19], Singularity [16,17,24], and Shifter [3,4].…”

Section: B Containerizationmentioning

confidence: 99%

Shared-Memory Communication for Containerized Workflows

Hobson

Yildiz

Nicolae

et al. 2021

2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid)

View full text Add to dashboard Cite

Scientific computation increasingly consists of a workflow of interrelated tasks. Containerization can make workflow systems more manageable, reproducible, and portable, but containers can impede communication due to their focus on encapsulation. In some circumstances, shared-memory regions are an effective way to improve performance of workflows; however sharing memory between containerized workflow tasks is difficult. In this work, we have created a software library called Dhmem that manages shared memory between workflow tasks in separate containers, with minimal code change and performance overhead. Instead of all code being in the same container, Dhmem allows a separate container for each workflow task to be constructed completely independently. Dhmem enables additional functionality: easy integration in existing workflow systems, communication configuration at runtime based on the environment, and scalable performance.

show abstract

“…Under this implementation, Kovács tested SysBench and IPerf (a network benchmark) using a single node (16 cores), finding CPU "almost at the level of the native performance" and network that "closely approximate[s] the performance of the native execution" [10]. Le and Paz tested MPI benchmarks on an unspecified number of nodes and a neuron simulation up to 8 nodes (192 cores), finding only "a small margin" for MPI but "more overhead" for the application [11]. Younge et al tested HPGC and an MPI benchmark up to 32 nodes (768 cores), finding that Singularity "does not impact network performance" and "provide[s] native performance" for HPCG [5].…”

Section: Related Workmentioning

confidence: 99%

HPC Container Runtimes have Minimal or No Performance Impact

Torrez

Randles

Priedhorsky

2019

2019 IEEE/ACM International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HP

View full text Add to dashboard Cite

HPC centers are facing increasing demand for greater software flexibility to support faster and more diverse innovation in computational scientific work. Containers, which use Linux kernel features to allow a user to substitute their own software stack for that installed on the host, are an increasingly popular method to provide this flexibility. Because standard container technologies such as Docker are unsuitable for HPC, three HPC-specific technologies have emerged: Charliecloud, Shifter, and Singularity. A common concern is that containers may introduce performance overhead. To our knowledge, no comprehensive, rigorous, HPC-focused assessment of container performance has previously been performed. Our present experiment compares the performance of all three HPC container implementations and bare metal on multiple dimensions using industry-standard benchmarks (SysBench, STREAM, and HPCG). We found no meaningful performance differences between the four environments, with the possible exception of modest variation in memory usage. These results suggest that HPC users should feel free to containerize their applications without concern about performance degradation, regardless of the container technology used. It is an encouraging development towards greater adoption of userdefined software stacks to increase the flexibility of HPC systems.

show abstract

Performance Analysis of Applications using Singularity Container on SDSC Comet

Cited by 26 publications

References 1 publication

CLASSIC v1.0: the open-source community successor to the Canadian Land Surface Scheme (CLASS) and the Canadian Terrestrial Ecosystem Model (CTEM) – Part 1: Model framework and site-level performance

CLASSIC v1.0: the open-source community successor to the Canadian Land Surface Scheme (CLASS) and the Canadian Terrestrial Ecosystem Model (CTEM) – Part 1: Model framework and site-level performance

Shared-Memory Communication for Containerized Workflows

HPC Container Runtimes have Minimal or No Performance Impact

Contact Info

Product

Resources

About