Zest Checkpoint storage system for large supercomputers

Nowoczynski, Paul; Stone, N. T. B.; Yanovich, Jared; Sommerfield, Jason

doi:10.1109/pdsw.2008.4811883

Cited by 22 publications

(19 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pylon is modeled on related PSC production filesystems [12][13] [14], including the shared filesystem on PSC's Data Exacell. Recognizing the need for both low latency access and long-term data safety, this system is entirely disk-based (reducing access time by a factor of ~10,000 compared to tape) but with highlevel RAID providing data safety appropriate to storage for at least the life of the Bridges project.…”

Section: Shared Parallel Filesystemmentioning

confidence: 99%

Bridges

Nystrom

Levine

Roskies

et al. 2015

Proceedings of the 2015 XSEDE Conference on Scientific Advancements Enabled by Enhanced Cyberinfrastructure - XSEDE '15

177

View full text Add to dashboard Cite

In this paper, we describe Bridges, a new HPC resource that will integrate advanced memory technologies with a uniquely flexible, user-focused, data-centric environment to empower new research communities, bring desktop convenience to HPC, connect to campuses, and drive complex workflows. Bridges will differ from traditional HPC systems and support new communities through extensive interactivity, gateways (convenient web interfaces that hide complex functionality and ease access to HPC resources) and tools for gateway building, persistent databases and web servers, high-productivity programming languages, and virtualization. Bridges will feature three tiers of processing nodes having 128GB, 3TB, and 12TB of hardware-enabled coherent shared memory per node to support memory-intensive applications and ease of use, together with persistent database and web nodes and nodes for logins, data transfer, and system management. State-of-the-art Intel ® Xeon ® CPUs and NVIDIA Tesla GPUs will power Bridges' compute nodes. Multiple filesystems will provide optimal handling for different data needs: a high-performance, parallel, shared filesystem, node-local filesystems, and memory filesystems. Bridges' nodes and parallel filesystem will be interconnected by the Intel Omni-Path Fabric, configured in a topology developed by PSC to be optimal for the anticipated data-centric workload. Bridges will be a resource on XSEDE, the NSF Extreme Science and Engineering Discovery Environment, and will interoperate with other advanced cyberinfrastructure resources. Through a pilot project with Temple University, Bridges will develop infrastructure and processes for campus bridging, consisting of offloading jobs at periods of unusually high load to the other site and facilitating cross-site data management. Education, training, and outreach activities will raise awareness of Bridges and data-intensive science across K-12 and university communities, industry, and the general public.

show abstract

Section: Shared Parallel Filesystemmentioning

confidence: 99%

Bridges

Nystrom

Levine

Roskies

et al. 2015

Proceedings of the 2015 XSEDE Conference on Scientific Advancements Enabled by Enhanced Cyberinfrastructure - XSEDE '15

177

View full text Add to dashboard Cite

show abstract

“…The development of middleware layers such as the Parallel Log-structured File System (PLFS) [4] and Zest [14] has, to an extent, bridged the gap between processor and I/O performance. In these systems multiple parallel writes are written sequentially to the file system with a log tracking the current data.…”

Section: Related Workmentioning

confidence: 99%

Computer Performance Engineering

Thomas¹

2011

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Input/output (I/O) operations can represent a significant proportion of the run-time when large scientific applications are run in parallel. Although there have been advances in the form of file-format libraries, file system design and I/O hardware, a growing divergence exists between the performance of parallel file systems and compute processing rates.In this paper we utilise RIOT, an input/output tracing toolkit being developed at the University of Warwick, to assess the performance of three standard industry I/O benchmarks and mini-applications. We present a case study demonstrating the tracing and analysis capabilities of RIOT at scale, using MPI-IO, Parallel HDF-5 and MPI-IO augmented with the Parallel Log-structured File System (PLFS) middle-ware being developed by the Los Alamos National Laboratory.

show abstract

“…In the Zest implementation of a log-structured file system [12], the data is written in this manner (via the fastest available path) to a temporary staging area that has no readback capability. This serves as a transition layer, caching data that is later copied to a fully featured file system at a noncritical time.…”

Section: Introductionmentioning

confidence: 99%

LDPLFS: Improving I/O Performance without Application Modification

Wright

Hammond

Pennycook

et al. 2012

2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops &Amp; PhD Forum

View full text Add to dashboard Cite

Abstract-Input/Output (I/O) operations can represent a significant proportion of run-time when large scientific applications are run in parallel and at scale. In order to address the growing divergence between processing speeds and I/O performance, the Parallel Log-structured File System (PLFS) has been developed by EMC Corporation and the Los Alamos National Laboratory (LANL) to improve the performance of parallel file activities. Currently, PLFS requires the use of either (i) the FUSE Linux Kernel module; (ii) a modified MPI library with a customised ROMIO MPI-IO library; or (iii) an application rewrite to utilise the PLFS API directly.In this paper we present an alternative method of utilising PLFS in applications. This method employs a dynamic library to intercept the low-level POSIX operations and retarget them to use the equivalents offered by PLFS. We demonstrate our implementation of this approach, named LDPLFS, on a set of standard UNIX tools, as well on as a set of standard parallel I/O intensive mini-applications. The results demonstrate almost equivalent performance to a modified build of ROMIO and improvements over the FUSE-based approach. Furthermore, through our experiments we demonstrate decreased performance in PLFS when ran at scale on the Lustre file system.

show abstract

Zest Checkpoint storage system for large supercomputers

Cited by 22 publications

References 2 publications

Bridges

Bridges

Computer Performance Engineering

LDPLFS: Improving I/O Performance without Application Modification

Contact Info

Product

Resources

About