The Technologies Required for Fusing HPC and Real-Time Data to Support Urgent Computing

Gibb, Gordon; Nash, Rupert W.; Brown, Nick; Prodan, Bianca

doi:10.1109/urgenthpc49580.2019.00009

Cited by 13 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Urgent analytics differ from other types of analytics because of the distribution and nature of the data, the complexity of the models, the strict error thresholds, and the time constraints. These workflows' execution must combine the demand for a significant amount of processing power to reduce errors with the requirement to make sure that crucial data streams are processed and assimilated promptly [3], [4].…”

Section: A Urgent Computingmentioning

confidence: 99%

“…The tremendous unpredictability of data, resources, and services throughout the Edge/Cloud-HPC continuum amplifies this heterogeneity. The SAGE catalog of plugins proposes applied models produced regardless of the device or runtime scenario in which they will be employed 4 . These plugins' scheduling does not account for customized behaviour when data content includes time-sensitive instructions or extra calculations.…”

Section: F Proof Of Conceptmentioning

confidence: 99%

See 1 more Smart Citation

Towards a Methodology for Building Dynamic Urgent Applications on Continuum Computing Platforms

Balouek-Thomert

Lefèvre

Caron

et al. 2022

2022 First Combined International Workshop on Interactive Urgent Supercomputing (CIW-IUS)

View full text Add to dashboard Cite

Advanced cyberinfrastructure aims at making the use of streaming data a common practice in the scientific community. They offer an ecosystem that links data, compute, network, and users to deliver knowledge obtained from multiple data sources using large-scale computational models. However, integrating this heterogeneous data with time-sensitive systems is difficult due to a lack of programming abstractions that can allow data-driven reactive behaviors throughout the edge-tocloud/HPC computing continuum. Here we present a methodology for incorporating contextual information into the application logic while taking into consideration the heterogeneity of the underlying platform and the unpredictability of the data. A fire science scenario that includes sensors at the network's edge for smoke detection and computational models launched in the cloud for wildfire simulation and air quality assessment serves as the inspiration for this method. We then discuss research directions for tackling similar scenarios with a particular focus on resource management and programming models.

show abstract

Section: A Urgent Computingmentioning

confidence: 99%

Section: F Proof Of Conceptmentioning

confidence: 99%

Towards a Methodology for Building Dynamic Urgent Applications on Continuum Computing Platforms

Balouek-Thomert

Lefèvre

Caron

et al. 2022

2022 First Combined International Workshop on Interactive Urgent Supercomputing (CIW-IUS)

View full text Add to dashboard Cite

show abstract

“…Python also lowers the difficulty for developers writing workflow stages, as it is a well known language, and so developers are likely already familiar with it. A more in-depth discussion on the language choices for the workflow manager and VESTEC system in general can be found at [10].…”

Section: A Languagementioning

confidence: 99%

“…In this section we describe how the workflow manager integrates with the other components of the system. The design of the system is presented in detail in [10].…”

Section: Integration With the Vestec Systemmentioning

confidence: 99%

“…VESTEC's design architecture, covered in detail in [10], consists of a centralised control system (running on a server), which has access to multiple HPC machines for running simulations, a web API to allow urgent decision makers to connect to the system, and functionality for acquiring real-time data from sources such as satellites and sensors. The control system adopts a modular approach, whereby each function it has to perform is represented by a single component, and the components work together to make up the full control system.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Bespoke Workflow Management System for Data-Driven Urgent HPC

Gibb

Brown

Nash

et al. 2020

2020 IEEE/ACM HPC for Urgent Decision Making (UrgentHPC)

Self Cite

View full text Add to dashboard Cite

In this paper we present a workflow management system which permits the kinds of data-driven workflows required by urgent computing, namely where new data is integrated into the workflow as a disaster progresses in order refine the predictions as time goes on. This allows the workflow to adapt to new data at runtime, a capability that most workflow management systems do not possess. The workflow management system was developed for the EU-funded VESTEC project, which aims to fuse HPC with real-time data for supporting urgent decision making. We first describe an example workflow from the VESTEC project, and show why existing workflow technologies do not meet the needs of the project. We then go on to present the design of our Workflow Management System, describe how it is implemented into the VESTEC system, and provide an example of the workflow system in use for a test case.

show abstract

Predicting accurate batch queue wait times on production supercomputers by combining machine learning techniques

Brown,

Gibb,

Belikov

et al. 2024

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

The ability to accurately predict when a job on a supercomputer will leave the queue and start to run is not only beneficial for providing insights to users, but can also help enable non‐traditional HPC workloads that are not necessarily suited to the batch queue style‐approach that is ubiquitous on production HPC machines. However there are numerous challenges in achieving such a prediction with high accuracy, not least because the queue's state can change rapidly and depend upon many factors. In this work, we explore a novel machine learning approach for predicting queue wait times, hypothesising that such a model can capture the complex behavior resulting from the queue policy and other interactions to generate accurate job start times. For ARCHER2 (HPE Cray EX), Cirrus (HPE 8600), and 4‐cabinet (HPE Cray EX) we explore how different machine learning approaches and techniques improve the accuracy of our predictions, comparing against the estimation generated by Slurm. By combining categorization and regression models, we demonstrate that our approach delivers the most accurate predictions across our machines of interest, with the result of this work being the ability to predict job start times within 1 min of the actual start time for around 65% of jobs on ARCHER2 and 4‐cabinet, and 76% of jobs on Cirrus. When compared against what Slurm can deliver, via the backfill plugin, this represents around 3.8 times better accuracy on ARCHER2 and 18 times better for Cirrus. Furthermore our approach can accurately predicting the start time for three quarters of all job within 10 min of the actual start time on ARCHER2 and 4‐cabinet, and for 90% of jobs on Cirrus. Whilst the initial driver of this work was to better facilitate non‐traditional, interactive and urgent, workloads on HPC machines, the insights gained can also be used to provide wider benefits to users, enrich existing batch queue systems, and inform supercomputing center policy also.

show abstract

The Technologies Required for Fusing HPC and Real-Time Data to Support Urgent Computing

Cited by 13 publications

References 11 publications

Towards a Methodology for Building Dynamic Urgent Applications on Continuum Computing Platforms

Towards a Methodology for Building Dynamic Urgent Applications on Continuum Computing Platforms

A Bespoke Workflow Management System for Data-Driven Urgent HPC

Predicting accurate batch queue wait times on production supercomputers by combining machine learning techniques

Contact Info

Product

Resources

About