Build and Execution Environment (BEE): an Encapsulated Environment Enabling HPC Applications Running Everywhere

Abstract

Variations in High Performance Computing (HPC) system software configurations mean that applications are typically configured and built for specific HPC environments. Building applications can require a significant investment of time and effort for application users and requires application users to have additional technical knowledge. Container technologies like Docker bring great benefits to the application development, build and deployment processes. While much cloud computing infrastructure is already designed to support Docker, little work has been done to support production Docker deployment on HPC systems. In this work, we propose a Docker-enabled Build and Execution Environment (BEE) for HPC systems and detail a standard backend for BEE using virtual machines, the BEE-VM. This brings many of the benefits of Docker to existing HPC machines in user-space without the requirement of specialized pre-installed software and with no system administrator configuration. We show that current HPC application can be easily configured to run within BEE, eliminating the need to reconfigure and rebuild applications for different systems while preserving comparable performance.

Generating and Scaling a Multi-Language Test-Suite for MPI

“…JUBE [14] focuses on running and analyzing benchmarks in a systematic way, with the ability to adapt to the heterogeneous systems through a minimal set of information in XML or YAML-based format. Beeswarm [18] [4] is a solution designed to automate application workflows and integrate with multiple CI providers, based on BEE [3], an autonomous containerization environment and workflow orchestration system. Unlike PCVS, it is not intended for end users but rather for programmatic use in automating the deployment of reproducible applications for validating production developments.…”

Enabling EASEY Deployment of Containerized Applications for Future HPC Systems

“…The authors of Build and Execution Environment BEE in [5] propose an execution framework which can, besides others, also deploy Charliecloud container on a HPC infrastructure. Their approach focuses on a variety of different cloud and cluster environments managed by the same authority.…”

Constructing Reliable Computing Environments on Top of Amazon EC2 Spot Instances

“…As containers on a host share the underlying operating system kernel, container migration is much more lightweight than a VM migration. Previous works have shown the effectiveness of said tiering approach (i.e., running containers within VM instances) in terms of performance [37], and as a first solution with which to implement fault-tolerance mechanism to run the applications on spot VMs [24]. Moreover, in [38][39][40] it is shown that migration of containers running various real applications is accomplished within a few tens of seconds at most.…”