On the role of burst buffers in leadership-class storage systems

Liu, Ning; Cope, Jason M.; Carns, Philip; Carothers, Christopher D.; Ross, Robert; Grider, Gary; Crume, Adam; Maltzahn, Carlos

doi:10.1109/msst.2012.6232369

Cited by 279 publications

(158 citation statements)

References 27 publications

Supporting

Mentioning

156

Contrasting

Order By: Relevance

“…[20] Bent et al [4] argue that the intermediate I/O nodes are the best place for burst buffers, since this choice allows to pipeline computation with data movement to servers. The same approach is evaluated by Liu et al [58] through simulation.…”

Section: Caching and Prefetchingmentioning

confidence: 99%

“…Darshan instruments POSIX, MPI-IO, HDF5, and PnetCDF libraries, providing data for a wide range of applications. Liu et al [58] …”

Section: Postmortem Analysismentioning

confidence: 99%

“…Liu et al [58] integrate an analytical model of burst buffers in the CODES storage system simulator in order to investigate the performance impact of adopting them in the storage infrastructure. Burst buffers data access costs are modeled with device throughput and latency.…”

Section: Performance Modelingmentioning

confidence: 99%

See 2 more Smart Citations

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

We present a comprehensive survey on parallel I/O in the high performance computing (HPC) context. This is an important field for HPC because of the historic gap between processing power and storage latencies, which causes applications performance to be impaired when accessing or generating large amounts of data. As the available processing power and amount of data increase, I/O remains a central issue for the scientific community. In this survey, we focus on a traditional I/O stack, with a POSIX parallel file system. We present background concepts everyone could benefit from. Moreover, through the comprehensive study of publications from the most important conferences and journals in a five-year time window, we discuss the state of the art of I/O optimization approaches, access pattern extraction techniques, and performance modeling, in addition to general aspects of parallel I/O research. Through this approach, we aim at identifying the general characteristics of the field and the main current and future research topics.

show abstract

Section: Caching and Prefetchingmentioning

confidence: 99%

“…Darshan instruments POSIX, MPI-IO, HDF5, and PnetCDF libraries, providing data for a wide range of applications. Liu et al [58] …”

Section: Postmortem Analysismentioning

confidence: 99%

See 1 more Smart Citation

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Typical unpacking destinations include tmpfs, compute node local storage, or a shared scratch filesystem, perhaps leveraging the trend toward close-to-node storage such as burst buffers [26]. Unpacking can take place during job allocation time or as the first step in the user's job script.…”

Section: Standard Reproducible Workflow (G1)mentioning

confidence: 99%

Charliecloud

Priedhorsky

Randles

2017

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Supercomputing centers are seeing increasing demand for userdefined software stacks (UDSS), instead of or in addition to the stack provided by the center. These UDSS support user needs such as complex dependencies or build requirements, externally required configurations, portability, and consistency. The challenge for centers is to provide these services in a usable manner while minimizing the risks: security, support burden, missing functionality, and performance. We present Charliecloud, which uses the Linux user and mount namespaces to run industry-standard Docker containers with no privileged operations or daemons on center resources. Our simple approach avoids most security risks while maintaining access to the performance and functionality already on offer, doing so in just 800 lines of code. Charliecloud promises to bring an industry-standard UDSS user workflow to existing, minimally altered HPC resources.

show abstract

“…An example being the CODES I/O language [8], which has been used to demonstrate performance improvements of burst buffer systems for some user interpreted workloads [9].…”

Section: Related Workmentioning

confidence: 99%

Replicating HPC I/O Workloads with Proxy Applications

Dickson

Wright

Maheswaran

et al. 2016

2016 1st Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems (PDSW-DISCS)

View full text Add to dashboard Cite

Abstract-Large scale simulation performance is dependent on a number of components, however the task of investigation and optimization has long favored computational and communication elements above I/O. Manually extracting the pattern of I/O behavior from a parent application is a useful way of working to address performance issues on a per-application basis, but developing workflows with some degree of automation and flexibility provides a more powerful approach to tackling current and future I/O challenges. In this paper we describe a workload replication workflow that extracts the I/O pattern of an application and recreates its behavior with a flexible proxy application. We demonstrate how simple lightweight characterization can be translated to provide an effective representation of a physics application, and show how a proxy replication can be used as a tool for investigating I/O library paradigms.

show abstract

On the role of burst buffers in leadership-class storage systems

Cited by 279 publications

References 27 publications

A Checkpoint of Research on Parallel I/O for High-Performance Computing

A Checkpoint of Research on Parallel I/O for High-Performance Computing

Charliecloud

Replicating HPC I/O Workloads with Proxy Applications

Contact Info

Product

Resources

About