ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management

Godoy, William F.; Podhorszki, Norbert; Wang, Ruonan; Atkins, Chuck; Eisenhauer, Greg; Gu, Jianhua; Davis, Philip E.; Choi, Jong Youl; Germaschewski, K.; Huck, Kevin; Huebl, Axel; Kim, Mark; Kress, James; Kurç, Tahsin; Liu, Qing; Logan, Jeremy; Mehta, Kshitij; Ostrouchov, George; Parashar, Manish; Poeschel, Franz; Pugmire, David; Suchyta, E.; Takahashi, Keichi; Thompson, Nick; Tsutsumi, Seiji; Wan, Lipeng; Wolf, Matthew; Wu, Kesheng; Klasky, Scott

doi:10.1016/j.softx.2020.100561

Cited by 160 publications

(69 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SCORPIO is derived from the Parallel I/O library (Hartnett & Edwards, 2021) and continues to support the same application programming interface. To improve the I/O write performance the library caches and rearranges output data between MPI processes before using low level I/O libraries like the netCDF, Parallel netCDF (PnetCDF) (Latham et al., 2003), and ADIOS (Godoy et al., 2020) libraries to write the data to the file system. On Cori the simulation produced ∼4.5 TB of data per simulated day and achieved an average I/O write throughput of ∼2.5 GB/s using the PnetCDF library.…”

Section: Performancementioning

confidence: 99%

Convection‐Permitting Simulations With the E3SM Global Atmosphere Model

Caldwell

Terai

Hillman

et al. 2021

J Adv Model Earth Syst

View full text Add to dashboard Cite

This paper describes the first implementation of the Δx = 3.25 km version of the Energy Exascale Earth System Model (E3SM) global atmosphere model and its behavior in a 40‐day prescribed‐sea‐surface‐temperature simulation (January 20 through February 28, 2020). This simulation was performed as part of the DYnamics of the Atmospheric general circulation Modeled On Non‐hydrostatic Domains (DYAMOND) Phase 2 model intercomparison. Effective resolution is found to be the horizontal dynamics grid resolution despite using a coarser grid for physical parameterizations. Despite this new model being in an immature and untuned state, moving to 3.25 km grid spacing solves several long‐standing problems with the E3SM model. In particular, Amazon precipitation is much more realistic, the frequency of light and heavy precipitation is improved, agreement between the simulated and observed diurnal cycle of tropical precipitation is excellent, and the vertical structure of tropical convection and coastal stratocumulus look good. In addition, the new model is able to capture the frequency and structure of important weather events (e.g., tropical cyclones, extratropical cyclones including atmospheric rivers, and cold air outbreaks). Interestingly, this model does not get rid of the erroneous southern branch of the intertropical convergence zone nor the tendency for strongest convection to occur over the Maritime Continent rather than the West Pacific, both of which are classic climate model biases. Several other problems with the simulation are identified, underscoring the fact that this model is a work in progress.

show abstract

Section: Performancementioning

confidence: 99%

Convection‐Permitting Simulations With the E3SM Global Atmosphere Model

Caldwell

Terai

Hillman

et al. 2021

J Adv Model Earth Syst

View full text Add to dashboard Cite

show abstract

“…(9). Furthermore, we used ADIOS2 package [30] along with MD simulations to reduce the time required for reading and analyzing data. For data presented in this paper we used a range of lattice size from 6 σ to 500 σ which led to different numbers of lattice points for our fixed simulation box.…”

Section: Definitions For Momentummentioning

confidence: 99%

Large Fluctuations in Nonideal Coarse-Grained Systems

Parsa¹,

Wagner²

2020

Phys. Rev. Lett.

View full text Add to dashboard Cite

At first glance the definition of mass and momentum appears to be uniquely defined. We show here, however, that this certainty can be misleading for many coarse grained systems. We show that particularly the fluctuating properties of common definitions of momentum in coarse grained methods like lattice gas and lattice Boltzmann do not agree with a fundamental definition of momentum. In the case of lattice gases, the definition of momentum will even disagree in the limit of large wavelength. For short times we derive analytical representations for the distribution of different momentum measures and thereby give a full account of these differences.

show abstract

“…To enable fast yet flexible data streaming over WANs, we utilize the ADIOS framework. 11,21 Traditionally, the ADIOS framework was utilized by simulation codes for fast I/O to filesystems on large HPC supercomputers. ADIOS also has "engine" protocols for remote, largescale, memory-to-memory transfer over long distances.…”

Section: Iiia Remote Streaming I/o With Adiosmentioning

confidence: 99%

A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

Churchill

Chang

Choi

et al. 2021

Fusion Science and Technology

View full text Add to dashboard Cite

The global nature of the ITER project along with its projected approximately petabyte-per-day data generation presents not only a unique challenge but also an opportunity for the fusion community to rethink, optimize, and enhance our scientific discovery process. Recognizing this, collaborative research with computational scientists was undertaken over the past several years to create a framework for large-scale data movement across wide-area networks to enable global near-real-time analysis of fusion data. This would broaden the available computational resources for analysis/simulation and increase the number of researchers actively participating in experiments.An official demonstration of this framework for fast, large data transfer and real-time analysis was carried out between the KSTAR tokamak in Daejeon, Korea, and Princeton Plasma Physics Laboratory (PPPL) in Princeton, New Jersey. Streaming large data transfer, with near-real-time movie creation and analysis of the KSTAR electron cyclotron emission imaging data, was performed using the Adaptable Input Output (I/O) System (ADIOS) framework, and comparisons were made at PPPL with simulation results from the XGC1 code. These demonstrations were made possible utilizing an optimized network configuration at PPPL, which achieved over 8.8 Gbps (88% utilization) in throughput tests from the National Fusion Research Institute to PPPL. This demonstration showed the feasibility for large-scale data analysis of KSTAR data and provides a nascent framework to enable use of globally distributed computational and personnel resources in pursuit of scientific knowledge from the ITER experiment.

show abstract

ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management

Cited by 160 publications

References 29 publications

Convection‐Permitting Simulations With the E3SM Global Atmosphere Model

Convection‐Permitting Simulations With the E3SM Global Atmosphere Model

Large Fluctuations in Nonideal Coarse-Grained Systems

A Framework for International Collaboration on ITER Using Large-Scale Data Transfer to Enable Near-Real-Time Analysis

Contact Info

Product

Resources

About