Storage 2020: A Vision for the Future of HPC Storage

Lockwood, GK; Hazen, Damian; Koziol, Quincey; Canon, RS; Antypas, Katie; Balewski, J.; Balthaser, N; Bhimji, Wahid; Botts, J; Broughton, Jeff; Butler, TL; Butler, GF; Cheema, Ravi; Daley, Christopher; Declerck, Tina; Gerhardt, L.; Hurlbert, WE; Kallback-Rose, KA; Leak, Stephen; Lee, J; Lee, R; Liu, J; Lozinskiy, Kirill; Paul, David; Prabhat,; Snavely, C.; Srinivasan, Jagan; Gibbins, T Stone; Wright, N. J.

doi:10.2172/1632124

Cited by 21 publications

(12 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a quest to achieve scalability and flexibility, HPC storage stacks have become increasingly heterogeneous [14]. In addition to parallel file systems, modern supercomputers feature a variety of additional storage subsystems (e.g., burst buffers [7] or key-value stores such as DAOS [16]) and deep memory hierarchies (HBM, DDRAM, NVM).…”

Section: Related Work and Positioningmentioning

confidence: 99%

“…As a consequence, the distributed nature of DNN model snapshots and the complex multi-level parallelism considerations make the problem of capturing and preserving/cloning intermediate DNN model snapshots non-trivial. This is further augmented by the need to adopt the FAIR principles and the increasingly complex heterogeneous storage stacks [14] that are deployed in modern HPC data centers. Nevertheless, there are also significant opportunities in this space: according to our previous study [13], the combination of data parallelism and layer-wise parallelism leads to subtle delays that can be exploited to to overlap the back-propagation with fine-grain asynchronous data management operations in the background, which can significantly reduce their overhead.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

DataStates: Towards Lightweight Data Models for Deep Learning

Nicolae

2020

Communications in Computer and Information Science

View full text Add to dashboard Cite

A key emerging pattern in deep learning applications is the need to capture intermediate DNN model snapshots and preserve or clone them to explore a large number of alternative training and/or inference paths. However, with increasing model complexity and new training approaches that mix data, model, pipeline and layer-wise parallelism, this pattern is challenging to address in a scalable and efficient manner. To this end, this position paper advocates for rethinking how to represent and manipulate DNN learning models. It relies on a broader notion of data states, a collection of annotated, potentially distributed data sets (tensors in the case of DNN models) that AI applications can capture at key moments during the runtime and revisit/reuse later. Instead explicitly interacting with the storage layer (e.g., write to a file), users can "tag" DNN models at key moments during runtime with metadata that expresses attributes and persistency/movement semantics. A highperformance runtime is the responsible to interpret the metadata and perform the necessary actions in the background, while offering a rich interface to find data states of interest. Using this approach has benefits at several levels: new capabilities, performance portability, high performance and scalability.

show abstract

Section: Related Work and Positioningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

DataStates: Towards Lightweight Data Models for Deep Learning

Nicolae

2020

Communications in Computer and Information Science

View full text Add to dashboard Cite

show abstract

“…In a quest to achieve scalability and flexibility, HPC storage stacks have become increasingly heterogeneous (Lockwood et al, 2017). In addition to parallel file systems (e.g., Lustre (Petersen, 2015)), modern supercomputers feature a variety of additional storage subsystems (e.g., burst buffers (Cao et al, 2017) or key-value stores such as DAOS (Lofstead et al, 2016)) and deep memory hierarchies (HBM, DDRAM, NVM).…”

Section: Introductionmentioning

confidence: 99%

Demystifying asynchronous I/O Interference in HPC applications

Tseng

Nicolae

Cappello

et al. 2021

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

With increasing complexity of HPC workflows, data management services need to perform expensive I/O operations asynchronously in the background, aiming to overlap the I/O with the application runtime. However, this may cause interference due to competition for resources: CPU, memory/network bandwidth. The advent of multi-core architectures has exacerbated this problem, as many I/O operations are issued concurrently, thereby competing not only with the application but also among themselves. Furthermore, the interference patterns can dynamically change as a response to variations in application behavior and I/O subsystems (e.g. multiple users sharing a parallel file system). Without a thorough understanding, I/O operations may perform suboptimally, potentially even worse than in the blocking case. To fill this gap, this paper investigates the causes and consequences of interference due to asynchronous I/O on HPC systems. Specifically, we focus on multi-core CPUs and memory bandwidth, isolating the interference due to each resource. Then, we perform an in-depth study to explain the interplay and contention in a variety of resource sharing scenarios such as varying priority and number of background I/O threads and different I/O strategies: sendfile, read/write, mmap/write underlining trade-offs. The insights from this study are important both to enable guided optimizations of existing background I/O, as well as to open new opportunities to design advanced asynchronous I/O strategies.

show abstract

“…They support only a single layer of storage devices [13]. Since each layer is an independent subsystem, users are forced to study each layer independently and make necessary adjustments in their applications [14].…”

Section: Introductionmentioning

confidence: 99%

MLBS: Transparent Data Caching in Hierarchical Storage for Out-of-Core HPC Applications

Alturkestani

Tonellot²,

Ltaief

et al. 2019

2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC)

View full text Add to dashboard Cite

Out-of-core simulation systems produce and/or consume a massive amount of data that cannot fit on a single compute node memory and that usually needs to be read and/or written back and forth during computation. I/O data movement may thus represent a bottleneck in large-scale simulations. To increase I/O bandwidth, high-end supercomputers are equipped with hierarchical storage subsystems such as node-local and remote-shared NVMe and SSD-based Burst Buffers. Advanced caching systems have recently been developed to efficiently utilize the multi-layered nature of the new storage hierarchy. Utilization of software components results in more efficient data accesses, at the cost of reduced computation kernel performance and limited numbers of simultaneous applications that can utilize the additional storage layers. We introduce MultiLayered Buffer Storage (MLBS), a data object container that provides novel methods for caching and prefetching data in out-of-core scientific applications to perform asynchronously expensive I/O operations on systems equipped with hierarchical storage. The main idea consists in decoupling I/O operations from computational phases using dedicated hardware resources to perform expensive context switches. MLBS monitors I/O traffic in each storage layer allowing fair utilization of shared resources while controlling the impact on kernels' performance. By continually prefetching up and down across all hardware layers of the memory/storage subsystems, MLBS transforms the original I/O-bound behavior of evaluated applications and shifts it closer to a memorybound regime. Our evaluation on a Cray XC40 system for a representative I/O-bound application, seismic inversion, shows that MLBS outperforms state-of-the-art filesystems, i.e., Lustre, Data Elevator and DataWarp by 6.06X, 2.23X, and 1.90X, respectively.

show abstract

Storage 2020: A Vision for the Future of HPC Storage

Cited by 21 publications

References 3 publications

DataStates: Towards Lightweight Data Models for Deep Learning

DataStates: Towards Lightweight Data Models for Deep Learning

Demystifying asynchronous I/O Interference in HPC applications

MLBS: Transparent Data Caching in Hierarchical Storage for Out-of-Core HPC Applications

Contact Info

Product

Resources

About