Automatizing the creation of specialized high-performance computing containers

Abstract

With Exascale computing already here, supercomputers are systems every time larger, more complex, and heterogeneous. While expert system administrators can install and deploy applications in the systems correctly, this is something that general users can not usually do. The eFlows4HPC project aims to provide methodologies and tools to enable the use and reuse of application workflows. One of the aspects that the project focuses on is simplifying the application deployment in large and complex systems. The approach uses containers, not generic ones, but containers tailored for each target High-Performance Computing (HPC) system. This paper presents the Container Image Creation service developed in the framework of the project and experimentation based on project applications. We compare the performance of the specialized containers against generic containers and against a native installation. The results show that in almost all cases, the specialized containers outperform the generic ones (up to 2× faster), and in all cases, the performance is the same as with the native installation.

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

“…The goal is that, given this topology and a HPC cluster location, it can be used by Yorc 5 (a TOSCA orchestrator) for deploying the software and data required to execute the workflow. The Software deployment is done by means of the Container Image Creation service which automates the creation of the container images for workflows, including the code as well as all the required software compiled for the target HPC platform [16]. On the other hand, the management of the required data is done by the Data Logistics Service 6 which executes the required data pipelines either at deployment or execution time.…”