A novel pilot job approach for improving the execution of distributed codes: application to the study of ordering in collisional transport in fusion plasmas

Abstract

SummaryThere are many applications in every scientific and engineering fields that have high computational requirements. They have to be adapted to either high‐performance computing or high‐throughput computing platforms, depending on their characteristics. For the latter case, the high volume of resources available in grid infrastructures should allow for a reduction of their calculation times and an increase of the accuracy of the results. Nevertheless, the low performance achieved by early‐binding methods is noticeable, and the current systems based on pilot jobs are not suitable for running legacy applications. Therefore, running a wide range of codes quickly and easily can be infeasible for many users and developers. For this reason, it would be of great advantage to count on pilot job techniques that truly exploit grid resources in an efficient way while maintaining compatibility with legacy applications. In this work, GWpilot is presented as a complete pilot job framework that overcomes the adaptability limitations of other pilot job systems to tackle the challenge of efficiently accomplishing high‐throughput calculations. To test the GWpilot system, two applications widely used for studying the properties of collisional transport in fusion plasmas (drift kinetic equation solver code and integrator of stochastic differential equations in plasmas) have been chosen as a proof of concept. Real calculations were performed with them to illustrate the adaptation process to the new framework and its performance. Additionally, the physics results obtained were analysed for completeness and further reading. Both applications also represent two different approaches further extended in the scientific community, that is, parameter sweep and Monte Carlo codes. Therefore, the general suitability of GWpilot for scientific and industrial applications belonging to other fields is demonstrated in this work. Copyright © 2014 John Wiley & Sons, Ltd.

Scheduling multiple virtual environments in cloud federations for distributed calculations

“…The GWpilot framework counts on two main modules in addition to the pilots: the GWpilot Server and the Factory. The behaviour of both is fully described in [5,48] and it is maintained in this work. On the other hand, the implementation of pilots is lightweight and without library dependencies, i.e.…”

“…The basic configuration of the framework is similar to the ones described in [5,48] but with two exceptions. The existence of a contextualisation file, which is not needed to be modified for a straightforward adaptation of any application, and the meaning of the −w option, that is now related to the maximum allowed time to boot the VM.…”

A simple model to exploit reliable algorithms in cloud federations

“…The GridWay Scheduler will use the information dynamically updated by the GWcloud Information driver to select the most suitable cloud provider every time that will execute a task. For this purpose, the Scheduler takes account of the requirements set in the description of the pilot jobs, in a way similar to the one the described in (Rubio- Montero et al, 2015a). Then, the management of the VM creation and the pilot job execution are delegated to the GWcloud Execution driver.…”

High‐performance computing: to boldly go where no human has gone before

“…On the other hand, Rubio-Montero et al [147] focus their work on a novel framework for pilot-job system based on GridWay meta-scheduler capable of improving high-throughput performance in the study of properties of collisional transport in fusion plasmas using grid available resources.…”