Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms

Merzky, André; Turilli, Matteo; Титов, М. И.; Al-Saadi, Aymen; Jha, Shantenu

doi:10.48550/arxiv.2103.00091

Cited by 2 publications

(2 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To support this feature the launcher relies on a combination of Slurm salloc/srun [15], or OAR containers [16]. For even more flexible schemes, we plan to support workflow pilot-based schedulers like Radical-Pilot [17] or QCG-PilotJob [18].…”

Section: Job Submission and Monitoringmentioning

confidence: 99%

A Framework for Large Scale Particle Filters Validated with Data Assimilation for Weather Simulation

Friedemann,

Keller,

et al. 2023

Preprint

View full text Add to dashboard Cite

Particle filters are a group of algorithms to solve inverse problems through statistical Bayesian methods when the model does not comply with the linear and Gaussian hypothesis. Particle filters are used in domains like data assimilation, probabilistic programming, neural network optimization, localization and navigation. Particle filters estimate the probability distribution of model states by running a large number of model instances, the so called particles. The ability to handle a very large number of particles is critical for high dimensional models. This paper proposes a novel paradigm to run very large ensembles of parallel model instances on supercomputers. The approach combines an elastic and fault tolerant runner/server model minimizing data movements while enabling dynamic load balancing. Particle weights are computed locally on each runner and transmitted when available to a server that normalizes them, resamples new particles based on their weight, and redistributes dynamically the work to runners to react to load imbalance. Our approach relies on a an asynchronously managed distributed particle cache permitting particles to move from one runner to another in the background while particle propagation goes on. This also enables the number of runners to vary during the execution either in reaction to failures and

show abstract

Section: Job Submission and Monitoringmentioning

confidence: 99%

A Framework for Large Scale Particle Filters Validated with Data Assimilation for Weather Simulation

Friedemann,

Keller,

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…RCT has three main components: RADICAL-SAGA (RS) [54], RADICAL-Pilot (RP) [55,56] and RADICAL EnTK [57,40]. RS is a Python implementation of the Open Grid Forum SAGA standard GFD.90 [58], a high-level interface to distributed infrastructure components like job schedulers, file transfer and resource provisioning services.…”

Section: Enabling Scalable Simulation Via the Radical Ensemble Toolkitmentioning

confidence: 99%

A Scalable Solution for Running Ensemble Simulations for Photovoltaic Energy

Hu¹,

Cervone²,

Turilli³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This chapter proposes and provides an in-depth discussion of a scalable solution for running ensemble simulation for solar energy production. Generating a forecast ensemble is computationally expensive. But with the help of Analog Ensemble, forecast ensembles can be generated with a single deterministic run of a weather forecast model. Weather ensembles are then used to simulate 11 10 KW photovoltaic solar power systems to study the simulation uncertainty under a wide range of panel configuration and weather conditions. This computational workflow has been deployed onto the NCAR supercomputer, Cheyenne, with more than 7,000 cores.Results show that, spring and summer are typically associated with a larger simulation uncertainty. Optimizing the panel configuration based on their individual performance under changing weather conditions can improve the simulation accuracy by more than 12%. This work also shows how panel configuration can be optimized based on geographic locations.

show abstract

Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms

Cited by 2 publications

References 34 publications

A Framework for Large Scale Particle Filters Validated with Data Assimilation for Weather Simulation

A Framework for Large Scale Particle Filters Validated with Data Assimilation for Weather Simulation

A Scalable Solution for Running Ensemble Simulations for Photovoltaic Energy

Contact Info

Product

Resources

About