Extreme Scale-out SuperMUC Phase 2 - lessons learned

Hammer, Nicolay; Jamitzky, Ferdinand; Satzger, Helmut; Allalen, Momme; Block, Alexander; Karmakar, Anupam; Brehm, Matthias; Bader, Reinhold; Iapichino, Luigi; Ragagnin, Antonio; Karakasis, Vasilios; Kranzlmüller, Dieter; Bode, Arndt; Huber, Herbert; Kühn, Martin; Machado, Rui; Grünewald, Daniel; Edelmann, Philipp V. F.; Röpke, Friedrich K.; Wittmann, Markus; Zeiser, Thomas; Wellein, Gerhard; Mathias, Gerald; Schwörer, Magnus; Lorenzen, Konstantin; Federrath, Christoph; Klessen, Ralf; Bamberg, Karl-Ulrich; Ruhl, Hartmut; Schornbaum, Florian; Bauer, Martin; Nikhil, Anand; Qi, Jiaxing; Klimach, Harald; Stüben, Hinnerk; Deshmukh, Abhishek; Falkenstein, Tobias; Dolag, Klaus; Petkova, Margarita

doi:10.48550/arxiv.1609.01507

Cited by 3 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More powerful supercomputers [1], [2] and advanced libraries [3], [4], [5], [6], [7] enable the training of ever more complex models on bigger data sets using advanced processing units such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs) at increasing speeds and efficiency. HPC hardware is advancing both through infrastructure of supercomputers, such as Fugaku [8], Summit [1] or the SuperMUC-NG [9], and through its components, such as TPU pods [2], specifically designed to ease large scale neural network training for users. Concurrent software improvements in form of more efficient libraries such as Horovod [6] allow executing general purpose code on large distributed clusters with minor code changes.…”

Section: Introductionmentioning

confidence: 99%

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Elnaggar

Heinzinger

Dallago

et al. 2020

Preprint

435

673

View full text Add to dashboard Cite

Motivation: Natural Language Processing (NLP) continues improving substantially through auto-regressive (AR) and auto-encoding (AE) Language Models (LMs). These LMs require expensive computing resources for self-supervised or un-supervised learning from huge unlabelled text corpora. The information learned is transferred through so-called embeddings to downstream prediction tasks. Computational biology and bioinformatics provide vast gold-mines of structured and sequentially ordered text data leading to extraordinarily successful protein sequence LMs that promise new frontiers for generative and predictive tasks at low inference cost. As recent NLP advances link corpus size to model size and accuracy, we addressed two questions: (1) To which extent can High-Performance Computing (HPC) up-scale protein LMs to larger databases and larger models? (2) To which extent can LMs extract features from single proteins to get closer to the performance of methods using evolutionary information? Methodology: Here, we trained two auto-regressive language models (Transformer-XL and XLNet) and two auto-encoder models (BERT and Albert) on 80 billion amino acids from 200 million protein sequences (UniRef100) and one language model (Transformer-XL) on 393 billion amino acids from 2.1 billion protein sequences taken from the Big Fat Database (BFD), today's largest set of protein sequences (corresponding to 22- and 112-times, respectively of the entire English Wikipedia). The LMs were trained on the Summit supercomputer, using 936 nodes with 6 GPUs each (in total 5616 GPUs) and one TPU Pod, using V3-512 cores. Results: We validated the feasibility of training big LMs on proteins and the advantage of up-scaling LMs to larger models supported by more data. The latter was assessed by predicting secondary structure in three- and eight-states (Q3=75-83, Q8=63-72), localization for 10 cellular compartments (Q10=74) and whether a protein is membrane-bound or water-soluble (Q2=89). Dimensionality reduction revealed that the LM-embeddings from unlabelled data (only protein sequences) captured important biophysical properties of the protein alphabet, namely the amino acids, and their well orchestrated interplay in governing the shape of proteins. In the analogy of NLP, this implied having learned some of the grammar of the language of life realized in protein sequences. The successful up-scaling of protein LMs through HPC slightly reduced the gap between models trained on evolutionary information and LMs. Additionally, our results highlighted the importance of bi-directionality when processing proteins as the uni-directional TransformerXL was outperformed by its bi-directional counterparts. Availability ProtTrans: <a href="https://github.com/agemagician/ProtTrans">https://github.com/agemagician/ProtTrans</a>

show abstract

Section: Introductionmentioning

confidence: 99%

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Elnaggar

Heinzinger

Dallago

et al. 2020

Preprint

435

673

View full text Add to dashboard Cite

show abstract

“…To clarify the last point, let's consider the case of a very large cosmological simulation that was run within the LRZ Extreme Scaling Workhop in 2015 [14]. Such simulation (Magneticum Box0/mr) had 1.2 • 10 7 particles per node, each node was allocating 4GB for the Barnes Hut tree, 22GB for the basic quantities used in gravity (e.g.…”

Section: Memory Transfermentioning

confidence: 99%

Gadget3 on GPUs with OpenACC

Ragagnin

Dolag

Wagner³

et al. 2020

Parallel Computing: Technology Trends

View full text Add to dashboard Cite

We present preliminary results of a GPU porting of all main Gadget3 modules (gravity computation, SPH density computation, SPH hydrodynamic force, and thermal conduction) using OpenACC directives. Here we assign one GPU to each MPI rank and exploit both the host and accellerator capabilities by overlapping computations on the CPUs and GPUs: while GPUs asynchronously compute interactions between particles within their MPI ranks, CPUs perform tree-walks and MPI communications of neighbouring particles. We profile various portions of the code to understand the origin of our speedup, where we find that a peak speedup is not achieved because of time-steps with few active particles. We run a hydrodynamic cosmological simulation from the Magneticum project, with 2·107 particles, where we find a final total speedup of ≈2. We also present the results of an encouraging scaling test of a preliminary gravity-only OpenACC porting, run in the context of the EuroHack17 event, where the prototype of the porting proved to keep a constant speedup up to 1024 GPUs.

show abstract

Gadget3 on GPUs with OpenACC

Ragagnin,

Dolag,

Wagner

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We present preliminary results of a GPU porting of all main Gad-get3 modules (gravity computation, SPH density computation, SPH hydrodynamic force, and thermal conduction) using OpenACC directives. Here we assign one GPU to each MPI rank and exploit both the host and accellerator capabilities by overlapping computations on the CPUs and GPUs: while GPUs asynchronously compute interactions between particles within their MPI ranks, CPUs perform treewalks and MPI communications of neighbouring particles. We profile various portions of the code to understand the origin of our speedup, where we find that a peak speedup is not achieved because of time-steps with few active particles. We run a hydrodynamic cosmological simulation from the Magneticum project, with 2 • 10 7 particles, where we find a final total speedup of ≈ 2. We also present the results of an encouraging scaling test of a preliminary gravity-only OpenACC porting, run in the context of the EuroHack17 event, where the prototype of the porting proved to keep a constant speedup up to 1024 GPUs.

show abstract

Extreme Scale-out SuperMUC Phase 2 - lessons learned

Abstract: We report lessons learned during the friendly user block operation period of the new system at the Leibniz Supercomputing Centre (SuperMUC Phase 2).

Cited by 3 publications

References 5 publications

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning

Gadget3 on GPUs with OpenACC

Gadget3 on GPUs with OpenACC

Contact Info

Product

Resources

About