Mochi: Composing Data Services for High-Performance Computing Environments

Ross, Robert; Amvrosiadis, George; Carns, Philip; Cranor, Charles D.; Dorier, Matthieu; Harms, Kevin; Ganger, Greg; Gibson, Garth A.; Gutiérrez, Samuel K.; Latham, Robert; Robey, Bob; Robinson, Dana; Settlemyer, Bradley W.; Shipman, Galen; Snyder, Shane A.; Soumagne, Jérôme; Zheng, Qing

doi:10.1007/s11390-020-9802-0

Cited by 60 publications

(16 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HEPnOS uses components from the Mochi project [1]. Thus it relies on Mercury [17] for remote procedure calls and remote direct memory access (RDMA), and on Argobots [18] for thread management.…”

Section: A Hepnos Storage Systemmentioning

confidence: 99%

“…For more technical details on how this mapping is done, we refer the reader to HEPnOS' extensive online documentation. 1 Of interest to the present work is the fact that some configuration parameters of HEPnOS are critical to its performance, including its number of database instances, how these databases map to threads, how it schedules its operations, down to low-level decisions such as whether to use blocking epoll or busy spinning for network progress.…”

Section: A Hepnos Storage Systemmentioning

confidence: 99%

“…Distributed data and input/output (I/O) services have become popular in high-performance computing (HPC) to replace traditional parallel file systems [1]. They range from multiuser, high-speed storage systems such as burst buffers [2], [3], [4], to transient, application-specific services providing processing capabilities such as in situ analysis [5], [6], [7].…”

Section: Introductionmentioning

confidence: 99%

“…While autotuning as a whole is a computationally expensive process, the similarity in the autotuning tasks presents opportunities for leveraging the knowledge gained from one autotuning campaign to the next. Examples include (1) the user deciding to increase the budget for tuning, but seeking to use the results from a number of smaller autotuning runs that were performed previously; (2) the user seeking to leverage autotuning results from a small scale to speed up the autotuning for largescale runs; or (3) the user introducing new parameters for autotuning but seeking to reuse the results for old parameters from previous runs.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

HPC Storage Service Autotuning Using Variational- Autoencoder -Guided Asynchronous Bayesian Optimization

Dorier

Égelé

Balaprakash

et al. 2022

2022 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

Distributed data storage services tailored to specific applications have grown popular in the high-performance computing (HPC) community as a way to address I/O and storage challenges. These services offer a variety of specific interfaces, semantics, and data representations. They also expose many tuning parameters, making it difficult for their users to find the best configuration for a given workload and platform.To address this issue, we develop a novel variationalautoencoder-guided asynchronous Bayesian optimization method to tune HPC storage service parameters. Our approach uses transfer learning to leverage prior tuning results and use a dynamically updated surrogate model to explore the large parameter search space in a systematic way.We implement our approach within the DeepHyper opensource framework, and apply it to the autotuning of a highenergy physics workflow on Argonne's Theta supercomputer. We show that our transfer-learning approach enables a more than 40× search speedup over random search, compared with a 2.5× to 10× speedup when not using transfer learning. Additionally, we show that our approach is on par with state-ofthe-art autotuning frameworks in speed and outperforms them in resource utilization and parallelization capabilities.

show abstract

Section: A Hepnos Storage Systemmentioning

confidence: 99%

Section: A Hepnos Storage Systemmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

HPC Storage Service Autotuning Using Variational- Autoencoder -Guided Asynchronous Bayesian Optimization

Dorier

Égelé

Balaprakash

et al. 2022

2022 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

show abstract

“…event tracing systems [16,34]. The idea of reusing idle CPU cycles to perform data computation can be further explored to enable more types of data operations [78].…”

Section: Processing Data Early Before Data Reaches Storagementioning

confidence: 99%

Streaming Data Reorganization at Scale with DeltaFS Indexed Massive Directories

et al. 2020

Self Cite

View full text Add to dashboard Cite

Complex storage stacks providing data compression, indexing, and analytics help leverage the massive amounts of data generated today to derive insights. It is challenging to perform this computation, however, while fully utilizing the underlying storage media. This is because, while storage servers with large core counts are widely available, single-core performance and memory bandwidth per core grow slower than the core count per die. Computational storage offers a promising solution to this problem by utilizing dedicated compute resources along the storage processing path. We present DeltaFS Indexed Massive Directories (IMDs), a new approach to computational storage. DeltaFS IMDs harvest available (i.e., not dedicated) compute, memory, and network resources on the compute nodes of an application to perform computation on data. We demonstrate the efficiency of DeltaFS IMDs by using them to dynamically reorganize the output of a real-world simulation application across 131,072 CPU cores. DeltaFS IMDs speed up reads by 1,740× while only slightly slowing down the writing of data during simulation I/O for in situ data processing.

show abstract

SOMA: Observability, monitoring, and in situ analytics for exascale applications

Yokelson,

Lappi,

Ramesh

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

SummaryWith the rise of exascale systems and large, data‐centric workflows, the need to observe and analyze high performance computing (HPC) applications during their execution is becoming increasingly important. HPC applications are typically not designed with online monitoring in mind, therefore, the observability challenge lies in being able to access and analyze interesting events with low overhead while seamlessly integrating such capabilities into existing and new applications. We explore how our service‐based observation, monitoring, and analytics (SOMA) approach to collecting and aggregating both application‐specific diagnostic data and performance data addresses these needs. We present our SOMA framework and demonstrate its viability with LULESH, a hydrodynamics proxy application. Then we focus on Astaroth, a multi‐GPU library for stencil computations, highlighting the integration of the TAU and APEX performance tools and SOMA for application and performance data monitoring.

show abstract

Mochi: Composing Data Services for High-Performance Computing Environments

Cited by 60 publications

References 30 publications

HPC Storage Service Autotuning Using Variational- Autoencoder -Guided Asynchronous Bayesian Optimization

HPC Storage Service Autotuning Using Variational- Autoencoder -Guided Asynchronous Bayesian Optimization

Streaming Data Reorganization at Scale with DeltaFS Indexed Massive Directories

SOMA: Observability, monitoring, and in situ analytics for exascale applications

Contact Info

Product

Resources

About