Parallel high-resolution climate data analysis using swift

Abstract

Advances in software parallelism and high-performance systems have resulted in an order of magnitude increase in the volume of output data produced by the Community Earth System Model (CESM). As the volume of data produced by CESM increases, the single-threaded script-based software packages traditionally used to post-process model output data have become a bottleneck in the analysis process. This paper presents a parallel version of the CESM atmosphere model data analysis workflow implemented using the Swift scripting language.Using the Swift implementation of the workflow, the time to analyze a 10-year atmosphere simulation on a typical cluster is reduced from 95 to 32 minutes on a single 8-core node and to 20 minutes on two nodes. The parallelized workflow is then used to evaluate several new data-intensive computational systems that feature RAM-based and flashbased storage. Even when constraining parallelism to limit the amount of file system space used by intermediate temporary data, our results show that the Swift-based implementation significantly reduces data analysis time.

A new end-to-end workflow for the Community Earth System Model (version 2.0) for the Coupled Model Intercomparison Project Phase 6 (CMIP6)

“…Each package requires different types of climatologies and plot types which creates unique performance characteristics for each of the packages. While previous efforts have enabled parallelism in the diagnostic packages (Woitaszek et al, 2011;Jacob et al, 2012), this approach resulted in poor performance for multiple file operations, and it had a steep learning curve for users. In order to create the climatology files in parallel and to reduce the expensive disk I/O operations, we developed the tool PyAverager (Paul et al, 2015;Mickelson et al, 2018).…”

A New End-to-End Workflow for the Community Earth System Model (version 2.0) for CMIP6

“…package requires different types of climatologies and plot types which creates unique performance characteristics for each of the packages. While previous efforts have enabled parallelism in the workflow (Woitaszek et al, November 2011;Jacob et al, 2012), this work presented it own set of issues. Specifically this approach resulting in poor performance for multiple file operations, and it had a steep learning curve for users.…”

A model for optimizing file access patterns using spatio-temporal parallelism

“…Woitaszek et al [14] gained performance by parallelizing a post-processing workflow for climate data using the Swift scripting language, with parallelism only scaling to 32 processes. On the other hand, our work is focused primarily on I/O performance scaling to thousands of processes.…”

“…Also, each individual file will be spatially split into more partitions. This will increase both the ns and pf terms in Equation 14. Thus using spatial parallelism will result in worse file access patterns as the number of processes grows.…”