Methodology for the Rapid Development of Scalable HPC Data Services

Abstract

There is growing evidence in the scientifc computing community that parallel file systems are not sufficient for all HPC storage workloads. This realization has motivated extensive research in new storage system designs. The question of which alternative we should turn towards implies that there could be a single answer satisfying a wide range of very diverse applications.We argue that such a generic solution does not exist. Instead, custom data services should be designed and tailored to the needs of specific applications on specific hardware. Furthermore, custom data services should be designed in close collaboration with users. In this paper, we present a methodology for the rapid development of such data services. This methodology promotes the design of reusable building blocks that can be composed together efficiently through a runtime based on high-performance threading, tasking, and remote procedure calls. We illustrate the success of our methodology by showcasing three examples of data services designed from the same building blocks, yet targeting entirely different applications.

Priority research directions for in situ data management: Enabling scientific discovery from diverse data sources

“…Individual NVRAM abstraction layers (e.g. libhio (Hjelm and Wright, 2017), Data Elevator (Dong et al, 2016), BurstMem (Wang et al, 2014), DataWarp (Henseler et al, 2016), Mochi (Carns et al, 2016;Dorier et al, 2018;Jenkins et al, 2017)) exist, but community-wide standards have yet to evolve.…”

A High-Performance Distributed Object-Store for Exascale Numerical Weather Prediction and Climate

“…In particular, accesses using enumerated range requests and wildcards, or exploring the stored data, require a more extensive index which, for these general purpose object stores, would require an external indexing service. We consider that an approach to building a domain-specific data service with associated APIs is likely to be more productive [5], and as such he MARS ecosystem and the FDB are built around operations driven by field-related metadata and the metadata schema.…”

Pufferscale: Rescaling HPC Data Services for High Energy Physics Applications

“…User-space data services are becoming increasingly popular in the HPC community as an alternative to traditional file-based methods for data storage and processing [1], [2], [3]. Although file-based approaches (e.g., POSIX-compliant parallel file systems) have certain advantages, such as standard interfaces with well-understood semantics, ensuring these properties leads to drawbacks including suboptimal underlying data storage organization and high overheads to preserve POSIX semantics unneeded by many HPC applications [4].…”

“…Although file-based approaches (e.g., POSIX-compliant parallel file systems) have certain advantages, such as standard interfaces with well-understood semantics, ensuring these properties leads to drawbacks including suboptimal underlying data storage organization and high overheads to preserve POSIX semantics unneeded by many HPC applications [4]. In contrast, user-space data services can be tailored to their target applications to meet specific needs [1]. They are usually deployed on compute nodes alongside the applications that use them.…”

“…An example is of such data service is HEPnOS [1], [5], a distributed storage system for High Energy Physics (HEP) applications developed in the context of the SciDAC4 "HEP on HPC" joint project between Argonne and Fermilab [6], based on a methodology and software components developed by the Mochi project [7]. HEP applications at FermiLab have traditionally used files stored on parallel file systems to manage event data coming out of particle accelerator and through pipelines of post-processing applications.…”