Scalable Data Management of the Uintah Simulation Framework for Next-Generation Engineering Problems with Radiation

Abstract

The need to scale next-generation industrial engineering problems to the largest computational platforms presents unique challenges. This paper focuses on data management related problems faced by the Uintah simulation framework at a production scale of 260K processes. Uintah provides a highly scalable asynchronous many-task runtime system, which in this work is used for the modeling of a 1000 megawatt electric (MWe) ultra-supercritical (USC) coal boiler. At 260K processes, we faced both parallel I/O and visualization related challenges, e.g., the default file-per-process I/O approach of Uintah did not scale on Mira. In this paper we present a simple to implement, restructuring based parallel I/O technique. We impose a restructuring step that alters the distribution of data among processes. The goal is to distribute the dataset such that each process holds a larger chunk of data, which is then written to a file independently. This approach finds a middle ground between two of the most common parallel I/O schemes-file per process I/O and shared file I/O-in terms of both the total number of generated files, and the extent of communication involved during the data aggregation phase. To address scalability issues when visualizing the simulation data, we developed a lightweight renderer using OSPRay, which allows scientists to visualize the data interactively at high quality and make production movies. Finally, this work presents a highly efficient and scalable radiation model based on the sweeping method, which significantly outperforms previous approaches in Uintah, like discrete ordinates. The integrated approach allowed the USC boiler problem to run on 260K CPU cores on Mira.

Node failure resiliency for Uintah without checkpointing

“…Users can choose a checkpointing interval and the type of I/O process among (1) I/O per process, (2) I/O per N processes (ie, one process collects output from N processes and performs I/O) and (3) using the PIDX high‐performance I/O library. For a small node count, I/O per process performs as well as PIDX on the Lustre file system, which is used by the Quartz cluster . For the modest size node counts used in this study, Uintah's built‐in I/O per process is used for checkpoint and restore and performs well in comparison to more sophisticated approaches such as PIDX …”

“…Comparison to high‐performance checkpointing : PIDX is a high‐performance I/O library built for high‐performance computing and performs better than the state‐of‐the‐art I/O libraries . Uintah with PIDX was able to utilize 80% of the theoretical disk bandwidth on MIRA . Compared with naive per‐process I/O, PIDX achieved a speed‐up of 2× to 10× as the number of processes increased from 8192 to 262 890 .…”

“…Uintah with PIDX was able to utilize 80% of the theoretical disk bandwidth on MIRA . Compared with naive per‐process I/O, PIDX achieved a speed‐up of 2× to 10× as the number of processes increased from 8192 to 262 890 . However, the speed‐up on Edison was limited to 0.7× to 1.3× as the number of cores increased from 1024 to 8192.…”

“…However, the speed‐up on Edison was limited to 0.7× to 1.3× as the number of cores increased from 1024 to 8192. This difference in performance occurs because Edison's Lustre file system performs better while handling large numbers of files compared with MIRA's GPFS file system . Charm++ has a resiliency capability using in‐memory checkpointing on the “buddy” process, which provided about 100× speed‐up over checkpointing .…”

Large‐eddy simulation of coal combustion in 15 MW pilot‐scale boiler‐simulation facility

“…A detailed multiscale multiphysics modeling running on high‐performance computing (HPC) or exascale computing facility could help researchers clear those hurdles for quantitative design and optimization of the combustion operation. Also, the recent growth in HPC has made it possible to use LES to simulate industrial‐scale engineering problems, for example, US DOE Exascale Computing project 13 …”