An elastic framework for ensemble-based large-scale data assimilation

Friedemann, Sebastian; Raffin, Bruno

doi:10.1177/10943420221110507

Cited by 7 publications

(3 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Intermediate models have been more recently explored. Members process data online relying on dynamic client/server 𝑁 × 𝑀 data communications [17,27,79]. These intermediate models keep the best of both worlds: the efficiency of a file-avoiding solution while retaining the necessary flexibility to support fault tolerance, load balancing and some elasticity.…”

Section: Simulation Ensemble Managementmentioning

confidence: 99%

High Throughput Training of Deep Surrogates from Large Ensemble Runs

Meyer,

Schouler,

Caulk

et al. 2023

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Recent years have seen a surge in deep learning approaches to accelerate numerical solvers, which provide faithful but computationally intensive simulations of the physical world. These deep surrogates are generally trained in a supervised manner from limited amounts of data slowly generated by the same solver they intend to accelerate. We propose an open-source framework that enables the online training of these models from a large ensemble run of simulations. It leverages multiple levels of parallelism to generate rich datasets. The framework avoids I/O bottlenecks and storage issues by directly streaming the generated data. A training reservoir mitigates the inherent bias of streaming while maximizing GPU throughput. Experiment on training a fully connected network as a surrogate for the heat equation shows the proposed approach enables training on 8TB of data in 2 hours with an accuracy improved by 47% and a batch throughput multiplied by 13 compared to a traditional offline procedure. CCS CONCEPTS• Computing methodologies → Online learning settings; Distributed artificial intelligence; Massively parallel and high-performance simulations; • Applied computing → Physical sciences and engineering.

show abstract

Section: Simulation Ensemble Managementmentioning

confidence: 99%

High Throughput Training of Deep Surrogates from Large Ensemble Runs

Meyer,

Schouler,

Caulk

et al. 2023

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…Which node the data is stored on and from which node the data is obtained are all transparent to the user. According to no need to care, the user only needs to manage the files and Distributed Processing System data as if they were operating the local file system [1] -2].…”

Section: Introductionmentioning

confidence: 99%

Development Process of Distributed System Based on Load Balancing Fault Tolerant Scheduling Algorithm

2022

DPS

View full text Add to dashboard Cite

With the rapid development of the Internet and the advancement of information technology, the server storage model is difficult to cope with the storage and management of data by companies or departments, and the file system becomes more and more complex. The purpose of this paper is to study the distributed system development process of fault-tolerant scheduling algorithm based on load balancing. This paper takes the distributed file system as the research object, points out the load skew problem in the process of load balancing, and analyzes the reason according to the file system architecture and design concept. Based on the concepts of static load balancing and dynamic load balancing, an optimal load balancing algorithm of FastDFS is proposed. The algorithm can not only ensure the load sharing ability of the system, improve the stability of the system, but also prevent the load imbalance in the process of linear expansion of the system, and overcome the original load balancing problem. Experiments show that the distributed system constructed in this paper has better resource scheduling performance, and the real-time load is also optimized to some extent.

show abstract

“…MelissaDA [116] In this chapter, we present the effortless framework protection leveraging Checkpoint-Restart (CR).…”

Section: Main Contributionsmentioning

confidence: 99%

Resilience for large ensemble computations

Keller¹

View full text Add to dashboard Cite

With the increasing power of supercomputers, ever more detailed models of physical systems can be simulated, and ever larger problem sizes can be considered for any kind of numerical system. During the last twenty years the performance of the fastest clusters went from the teraFLOPS domain (ASCI RED: 2.3 teraFLOPS) to the pre-exaFLOPS domain (Fugaku: 442 petaFLOPS), and we will soon have the first supercomputer with a peak performance cracking the exaFLOPS (El Capitan: 1.5 exaFLOPS). Ensemble techniques experience a renaissance with the availability of those extreme scales. Especially recent techniques, such as particle filters, will benefit from it. Current ensemble methods in climate science, such as ensemble Kalman filters, exhibit a linear dependency between the problem size and the ensemble size, while particle filters show an exponential dependency. Nevertheless, with the prospect of massive computing power come challenges such as power consumption and fault-tolerance. The mean-time-between-failures shrinks with the number of components in the system, and it is expected to have failures every few hours at exascale. In this thesis, we explore and develop techniques to protect large ensemble computations from failures. We present novel approaches in differential checkpointing, elastic recovery, fully asynchronous checkpointing, and checkpoint compression. Furthermore, we design and implement a fault-tolerant particle filter with pre-emptive particle prefetching and caching. And finally, we design and implement a framework for the automatic validation and application of lossy compression in ensemble data assimilation. Altogether, we present five contributions in this thesis, where the first two improve state-of-the-art checkpointing techniques, and the last three address the resilience of ensemble computations. The contributions represent stand-alone fault-tolerance techniques, however, they can also be used to improve the properties of each other. For instance, we utilize elastic recovery (2nd contribution) for mitigating resiliency in an online ensemble data assimilation framework (3rd contribution), and we built our validation framework (5th contribution) on top of our particle filter implementation (4th contribution). We further demonstrate that our contributions improve resilience and performance with experiments on various architectures such as Intel, IBM, and ARM processors. Amb l’increment de les capacitats de còmput dels supercomputadors, es poden simular models de sistemes físics encara més detallats, i es poden resoldre problemes de més grandària en qualsevol tipus de sistema numèric. Durant els últims vint anys, el rendiment dels clústers més ràpids ha passat del domini dels teraFLOPS (ASCI RED: 2.3 teraFLOPS) al domini dels pre-exaFLOPS (Fugaku: 442 petaFLOPS), i aviat tindrem el primer supercomputador amb un rendiment màxim que sobrepassa els exaFLOPS (El Capitan: 1.5 exaFLOPS). Les tècniques d’ensemble experimenten un renaixement amb la disponibilitat d’aquestes escales tan extremes. Especialment les tècniques més noves, com els filtres de partícules, se¿n beneficiaran. Els mètodes d’ensemble actuals en climatologia, com els filtres d’ensemble de Kalman, exhibeixen una dependència lineal entre la mida del problema i la mida de l’ensemble, mentre que els filtres de partícules mostren una dependència exponencial. No obstant, juntament amb les oportunitats de poder computar massivament, apareixen desafiaments com l’alt consum energètic i la necessitat de tolerància a errors. El temps de mitjana entre errors es redueix amb el nombre de components del sistema, i s’espera que els errors s’esdevinguin cada poques hores a exaescala. En aquesta tesis, explorem i desenvolupem tècniques per protegir grans càlculs d’ensemble d’errors. Presentem noves tècniques en punts de control diferencials, recuperació elàstica, punts de control totalment asincrònics i compressió de punts de control. A més, dissenyem i implementem un filtre de partícules tolerant a errors amb captació i emmagatzematge en caché de partícules de manera preventiva. I finalment, dissenyem i implementem un marc per la validació automàtica i l’aplicació de compressió amb pèrdua en l’assimilació de dades d’ensemble. En total, en aquesta tesis presentem cinc contribucions, les dues primeres de les quals milloren les tècniques de punts de control més avançades, mentre que les tres restants aborden la resiliència dels càlculs d’ensemble. Les contribucions representen tècniques independents de tolerància a errors; no obstant, també es poden utilitzar per a millorar les propietats de cadascuna. Per exemple, utilitzem la recuperació elàstica (segona contribució) per a mitigar la resiliència en un marc d’assimilació de dades d’ensemble en línia (tercera contribució), i construïm el nostre marc de validació (cinquena contribució) sobre la nostra implementació del filtre de partícules (quarta contribució). A més, demostrem que les nostres contribucions milloren la resiliència i el rendiment amb experiments en diverses arquitectures, com processadors Intel, IBM i ARM.

show abstract

An elastic framework for ensemble-based large-scale data assimilation

Cited by 7 publications

References 46 publications

High Throughput Training of Deep Surrogates from Large Ensemble Runs

High Throughput Training of Deep Surrogates from Large Ensemble Runs

Development Process of Distributed System Based on Load Balancing Fault Tolerant Scheduling Algorithm

Resilience for large ensemble computations

Contact Info

Product

Resources

About