Exascale Storage Systems - An Analytical Study of Expenses

Kunkel, Julian; Kuhn, Michael; Ludwig, Thomas

doi:10.14529/jsfi140106

Cited by 9 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DeltaFS [188] is a file system that improves the scalability of file systems by letting compute nodes manage metadata instead of a centralized server (common in traditional distributed file systems). Kunkel et al [105] review three methods to improve I/O performance at exascale: re-computation (found to be beneficial for infrequent accesses),…”

Section: I/o Middleware I/o Performance Can Also Be Improved By Solumentioning

confidence: 99%

The Landscape of Exascale Research

et al. 2020

View full text Add to dashboard Cite

The next generation of supercomputers will break the exascale barrier. Soon we will have systems capable of at least one quintillion (billion billion) floating-point operations per second (10 18 FLOPS). Tremendous amounts of work have been invested into identifying and overcoming the challenges of the exascale era. In this work, we present an overview of these efforts and provide insight into the important trends, developments, and exciting research opportunities in exascale computing. We use a three-stage approach in which we (1) discuss various exascale landmark studies, (2) use data-driven techniques to analyze the large collection of related literature, and (3) discuss eight research areas in depth based on influential articles. Overall, we observe that great advancements have been made in tackling the two primary exascale challenges: energy efficiency and fault tolerance. However, as we look forward, we still foresee two major concerns: the lack of suitable programming tools and the growing gap between processor performance and data bandwidth (i.e., memory, storage, networks). Although we will certainly reach exascale soon, without additional research, these issues could potentially limit the applicability of exascale computing.

show abstract

Section: I/o Middleware I/o Performance Can Also Be Improved By Solumentioning

confidence: 99%

The Landscape of Exascale Research

et al. 2020

View full text Add to dashboard Cite

show abstract

“…While lossless methods are often viewed as "safer" for scientific data, it is well known that lossless data compression of floating-point simulation data is difficult and often yields little benefit (e.g., Lindstrom and Isenburg, 2006;Bicer et al, 2013;Lakshminarasimhan et al, 2011). The reason for the relative ineffectiveness of lossless methods on scientific data (in contrast to image or audio data, for example) is that trailing digits of the fixed-precision floating-point output data are often essentially random, depending on the data type and the number of physically significant digits.…”

Section: Data Compressionmentioning

confidence: 99%

“…Random numbers are a liability for compression, thus giving lossy methods a significant advantage. Many recent efforts have focused on effectively applying or adapting lossy techniques for scientific datasets (e.g., Lakshminarasimhan et al, 2011;Iverson et al, 2012;Laney et al, 2013;Gomez and Cappello, 2013;Lindstrom, 2014). In the climate modeling community in particular, lossy data compression has been the subject of a number of recent studies (e.g., Woodring et al, 2011;Hübbe et al, 2013;Bicer et al, 2013;Baker et al, 2014;Kuhn et al, 2016;Silver and Zender, 2016;Zender, 2016), though we are not aware of comparable efforts on evaluating the effects on the scientific validity of the climate data and results.…”

Section: Data Compressionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating lossy data compression on climate simulation data within a large ensemble

et al. 2016

View full text Add to dashboard Cite

Abstract. High-resolution Earth system model simulations generate enormous data volumes, and retaining the data from these simulations often strains institutional storage resources. Further, these exceedingly large storage requirements negatively impact science objectives, for example, by forcing reductions in data output frequency, simulation length, or ensemble size. To lessen data volumes from the Community Earth System Model (CESM), we advocate the use of lossy data compression techniques. While lossy data compression does not exactly preserve the original data (as lossless compression does), lossy techniques have an advantage in terms of smaller storage requirements. To preserve the integrity of the scientific simulation data, the effects of lossy data compression on the original data should, at a minimum, not be statistically distinguishable from the natural variability of the climate system, and previous preliminary work with data from CESM has shown this goal to be attainable. However, to ultimately convince climate scientists that it is acceptable to use lossy data compression, we provide climate scientists

show abstract

“…However, the operator splitting approach of MESSy proves more powerful, also allowing for coupling of different domains, e.g. demonstrated by the integration of an ocean subsystem (Pozzer et al, 2011). An extension by Kerkweg and Jöckel (2012b) allowed for one-way coupling of different spatially nested domains using a server-client approach with point-to-point communication.…”

Section: Messymentioning

confidence: 99%

A diagnostic interface for the ICOsahedral Non-hydrostatic (ICON) modelling framework based on the Modular Earth Submodel System (MESSy v2.50)

Kern

Jöckel

2016

Geosci. Model Dev.

View full text Add to dashboard Cite

Abstract. Numerical climate and weather models have advanced to finer scales, accompanied by large amounts of output data. The model systems hit the input and output (I/O) bottleneck of modern high-performance computing (HPC) systems. We aim to apply diagnostic methods online during the model simulation instead of applying them as a post-processing step to written output data, to reduce the amount of I/O. To include diagnostic tools into the model system, we implemented a standardised, easy-to-use interface based on the Modular Earth Submodel System (MESSy) into the ICOsahedral Non-hydrostatic (ICON) modelling framework. The integration of the diagnostic interface into the model system is briefly described. Furthermore, we present a prototype implementation of an advanced online diagnostic tool for the aggregation of model data onto a user-defined regular coarse grid. This diagnostic tool will be used to reduce the amount of model output in future simulations. Performance tests of the interface and of two different diagnostic tools show, that the interface itself introduces no overhead in form of additional runtime to the model system. The diagnostic tools, however, have significant impact on the model system's runtime. This overhead strongly depends on the characteristics and implementation of the diagnostic tool. A diagnostic tool with high inter-process communication introduces large overhead, whereas the additional runtime of a diagnostic tool without inter-process communication is low. We briefly describe our efforts to reduce the additional runtime from the diagnostic tools, and present a brief analysis of memory consumption. Future work will focus on optimisation of the memory footprint and the I/O operations of the diagnostic interface.

show abstract

Exascale Storage Systems - An Analytical Study of Expenses

Cited by 9 publications

References 14 publications

The Landscape of Exascale Research

The Landscape of Exascale Research

Evaluating lossy data compression on climate simulation data within a large ensemble

A diagnostic interface for the ICOsahedral Non-hydrostatic (ICON) modelling framework based on the Modular Earth Submodel System (MESSy v2.50)

Contact Info

Product

Resources

About