A container-based workflow for distributed training of deep learning algorithms in HPC clusters

Abstract

Deep learning has been postulated as a solution for numerous problems in different branches of science. Given the resource-intensive nature of these models, they often need to be executed on specialized hardware such graphical processing units (GPUs) in a distributed manner. In the academic field, researchers get access to this kind of resources through High Performance Computing (HPC) clusters. This kind of infrastructures make the training of these models difficult due to their multi-user nature and limited user permission. In addition, different HPC clusters may possess different peculiarities that can entangle the research cycle (e.g., libraries dependencies). In this paper we develop a workflow and methodology for the distributed training of deep learning models in HPC clusters which provides researchers with a series of novel advantages. It relies on udocker as containerization tool and on Horovod as library for the distribution of the models across multiple GPUs. udocker does not need any special permission, allowing researchers to run the entire workflow without relying on any administrator. Horovod ensures the efficient distribution of the training independently of the deep learning framework used. Additionally, due to containerization and specific features of the workflow, it provides researchers with a cluster-agnostic way of running their models. The experiments carried out show that the workflow offers good scalability in the distributed training of the models and that it easily adapts to different clusters.

Using Explainability to Inform Statistical Downscaling Based on Deep Learning Beyond Standard Validation Approaches

“…In this nodes using one GPU, CNN-DeepESD training takes about 2 seconds per epoch, CNN-PAN 3 seconds and CNN-UNET 50 seconds. To ease the reproducibility of these experiments, as well as to simplify their execution on HPC clusters, we provide the required scripts to follow the workflow presented in (González-Abad, López García, & Kozlov, 2022). Dockerfiles are also available in the GitHub repository.…”

Using Explainability to Inform Statistical Downscaling Based on Deep Learning Beyond Standard Validation Approaches

“…We train the models in nodes equipped with graphical processing units (GPUs), more specifically NVIDIA Tesla V100 GPUs. In this nodes using one GPU, CNN‐DeepESD training takes about 2 s per epoch, CNN‐PAN 3 s and CNN‐UNET 50 s. To ease the reproducibility of these experiments, as well as to simplify their execution on HPC clusters, we provide the required scripts to follow the workflow presented in González‐Abad, López García, and Kozlov (2022)). Dockerfiles are also included with the code.…”