Yellowstone: A Dedicated Resource for Earth System Science

Loft, Richard; Andersen, Aaron; Bryan, Frank O.; Dennis, John M.; Engel, Tom; Gillman, Pam; Hart, David; Elahi, Irfan; Ghosh, Siddhartha; Kelly, Rory; Kamrath, Anke; Pfister, G.; Rempel, Matthias; Small, R. Justin; Skamarock, William C.; Wiltberger, M. J.; Shader, Bryan L.; Chen, Po; Cash, Benjamin A.

doi:10.1201/9781351103930-8

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These results were run on the NCAR-Wyoming Yellowstone supercomputer, for which the intra-node communication bandwidth has been measured to be 60GBps, while the peak one-way network speed (inter-node) is about 6GBps. 27 Note that the communication time for both levels is nearly identical for N = 16, when all MPI ranks are within a single shared memory node. Thus, the additional communication time of the first level of the FFT routine for cases with more than 16 time instances can be attributed to the non-local nature of this schedule, which results in increased inter-node versus intra-node communication as N increases.…”

Section: Iiia First-order Derivative Problem Resultsmentioning

confidence: 99%

“…with y = y initial + ∆y (27) Here [DD T S ] and [D T S ] are the matrices containing the time-spectral coefficients for the second and firstorder derivative terms of y, respectively, and y initial is the initial value for y. Applying the same idea used previously in the approximate factorization algorithm, ∆y can be found by taking equation (26) to the frequency domain.…”

Section: Iiib Second-order Derivative Problem Resultsmentioning

confidence: 99%

See 1 more Smart Citation

An Order N log ₂(N) Parallel Solver for Time Spectral Problems

Ramezanian

Mavriplis

2017

55th AIAA Aerospace Sciences Meeting

View full text Add to dashboard Cite

The time-spectral (TS) method is a fast and efficient scheme for computing the solution to temporal periodic problems. Compared to traditional backward difference timeimplicit methods, time-spectral methods incur significant computational savings by taking advantage of the temporal Fourier representation of the time discretization. However, the computational cost for current time-spectral solver implementations, which are based on the discrete Fourier transform (DFT), scales as O(N 2) where N represents the number of time instances. For parallel implementations, where each time instance is assigned to an individual processor, the wall clock time necessary to converge time-spectral solutions is of order O(N), whereas, the wall clock time for an optimal solver in this weak scaling approach is expected to be on the order of O(1). The present paper is focused on making the current TS method more efficient by reducing the computational cost to O(N log 2 N) for even and O(2N log 3 N) for odd numbers of samples (where N is a power of 3) and hence for parallel implementations, achieving a wall clock time on the order of O(log 2 N) for even and O(log 3 N) for odd numbers of samples. This is achieved by developing a parallel fast Fourier transform (FFT) implementation instead of the current discrete Fourier transform approach for evaluating and solving the temporal derivative terms, which significantly decreases the computational cost and results in notably shorter wall clock time.

show abstract

Section: Iiia First-order Derivative Problem Resultsmentioning

confidence: 99%

Section: Iiib Second-order Derivative Problem Resultsmentioning

confidence: 99%

An Order N log ₂(N) Parallel Solver for Time Spectral Problems

Ramezanian

Mavriplis

2017

55th AIAA Aerospace Sciences Meeting

View full text Add to dashboard Cite

show abstract

“…We first evaluate the performance of our new barotropic solver in CESM1.2.0 on the Yellowstone supercomputer, located at NCAR-Wyoming Supercomputing Center (NWSC) [23]. Yellowstone uses 2.6-Ghz Intel Xeon E5-2670 "Sandy Bridge" processors providing a total of 72,576 cores that are connected by a 13.6 GBps InfiniBand network.…”

Section: Resultsmentioning

confidence: 99%

Improving the scalability of the ocean barotropic solver in the community earth system model

Baker

Tseng

et al. 2015

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

High-resolution climate simulations are increasingly in demand and require tremendous computing resources. In the Community Earth System Model (CESM), the Parallel Ocean Model (POP) is computationally expensive for high-resolution grids (e.g., 0.1 •) and is frequently the least scalable component of CESM for certain production simulations. In particular, the modified Preconditioned Conjugate Gradient (PCG), used to solve the elliptic system of equations in the barotropic mode, scales poorly at the high core counts, which is problematic for high-resolution simulations. In this work, we demonstrate that the communication costs in the barotropic solver occupy an increasing portion of the total POP execution time as core counts are increased. To mitigate this problem, we implement a preconditioned Chebyshev-type iterative method in POP (called P-CSI), which requires far fewer global reductions than PCG. We also develop an effective block preconditioner based on the Error Vector Propagation Method to attain a competitive convergence rate for P-CSI. We demonstrate that the improved scalability of P-CSI results in a 5.2x speedup of the barotropic mode in high-resolution POP on 16,875 cores, which yields a 1.7x speedup of the overall POP simulation. Further, we ensure that the new solver produces an ocean climate consistent with the original one via an ensemble-based statistical method.

show abstract

“…The Yellowstone supercomputer is an IBM iDataPlex cluster built out of 4,536 dual socket Intel E5-2670 (Sandy Bridge) 16-core nodes interconnected using Mellanox FDR InfiniBand fabric in a full fat-tree topology [3]. There are 72 compute nodes in each of 63 racks.…”

Section: Yellowstone Supercomputermentioning

confidence: 99%

Application Performance Impact on Trimming of a Full Fat Tree InfiniBand Fabric

Ghosh¹,

DelVento²,

Kelly³

et al. 2016

2016 2nd IEEE International Workshop on High-Performance Interconnection Networks in the Exascale and Big-Data Era (HiPINEB)

Self Cite

View full text Add to dashboard Cite

We measured InfiniBand traffic in our full fat tree fabric and measured performance impact of trimming the fabric on our major application kernels. Based on traffic pattern analysis and application performance impact we infer that a 2:1 trimmed fat tree is a cost effective alternative to a full fat tree for this specific set of applications. The methodology we used may be useful for others who are performing design trade-offs for HPC systems. We also propose that switch hardware vendors design director class switches with trimmed fat tree options that optimize per port costs.

show abstract

Yellowstone: A Dedicated Resource for Earth System Science

Cited by 4 publications

References 0 publications

An Order N log ₂(N) Parallel Solver for Time Spectral Problems

An Order N log ₂(N) Parallel Solver for Time Spectral Problems

Improving the scalability of the ocean barotropic solver in the community earth system model

Application Performance Impact on Trimming of a Full Fat Tree InfiniBand Fabric

Contact Info

Product

Resources

About

Yellowstone: A Dedicated Resource for Earth System Science

Cited by 4 publications

References 0 publications

An Order N log 2(N) Parallel Solver for Time Spectral Problems

An Order N log 2(N) Parallel Solver for Time Spectral Problems

Improving the scalability of the ocean barotropic solver in the community earth system model

Application Performance Impact on Trimming of a Full Fat Tree InfiniBand Fabric

Contact Info

Product

Resources

About

An Order N log ₂(N) Parallel Solver for Time Spectral Problems

An Order N log ₂(N) Parallel Solver for Time Spectral Problems