CUBE: A scalable framework for large-scale industrial simulations

Abstract

Writing high performance solvers for engineering applications is a delicate task. These codes are often developed on an application to application basis, highly optimized to solve a certain problem. Here, we present our work on developing a general simulation framework for efficient computation of time resolved approximations of complex industrial flow problems -Complex Unified Building cubE method (Cube). To address the challenges of emerging, modern supercomputers, suitable data structures and communication patterns are developed and incorporated into Cube. We use a Cartesian grid together with various immersed boundary methods to accurately capture moving, complex geometries. The asymmetric workload of the immersed boundary is balanced by a predictive dynamic load balancer, and a multithreaded halo-exchange algorithm is employed to efficiently overlap communication with computations. Our work also concerns efficient methods for handling the large amount of data produced by large-scale flow simulations, such as scalable parallel I/O, data compression and in-situ processing.

Full Eulerian deformable solid‐fluid interaction scheme based on building‐cube method for large‐scale parallel computing

“…We use hybrid MPI/OpenMP parallel programming techniques for the parallelization, where whole cubes are subdivided between MPI ranks and thread parallelization of the numerical kernels (loop processing for spatial index) are performed on whole cube basis on the cubes within an MPI rank. In addition, MPI partitioning is performed by the following equation in order that the distribution of computational load on each MPI rank becomes uniform.…”

Digital transformation of droplet/aerosol infection risk assessment realized on "Fugaku" for the fight against COVID-19

“…We have applied the novel CFD method and software CUBE [8], which while being an efficient and scalable solver being based on explicit methods, but with a number of new provisions to allow it to be applied to real industrial applications much like the mainstream unstructured solvers, in fact being sufficiently accurate modeling of complex and difficult boundary conditions such as car window panes at extreme high resolutions required for modeling the detailed analysis of aerosols in real societal situations. Moreover, being based on explicit methods, mesh generations from 'dirty' CAD data can be effectively automated, in a matter of minutes instead of weeks typically required for unstructured models, allowing generations of multitudes of 'digital twins' representing accurate renditions of complex societal situations of classrooms, auditoriums, trains, busses, planes, restaurants etc.…”

“…

The fastest supercomputer in 2020, Fugaku, has not only achieved digital transformation of epidemiology in allowing end-to-end, detailed quantitative modeling of COVID-19 transmissions for the first time, but also transformed the behavior of the entire Japanese public through its detailed analysis of transmission risks in multitudes of societal situations entailing heavy risks. A novel aerosol simulation methodology was synthesized out of a combination of a new CFD methods meeting industrial demands, CUBE [8], which not only allowed the simulations to scale massively with high resolution required for micrometer virus-containing aerosol particles, but also extremely rapid time-to-solution due to its ability to generate the digital twins representing multitudes of societal situations in minutes not week, attaining true overall application high performance; such simulations have been running for the past 1.5 years on Fugaku, cumulatively consuming top supercomputer-class resources and the result communicated by the media as well as becoming official public policies. Keywords COVID-19 • Computational fluid dynamics • Building cube method • Immersed boundary method • Dirty CAD • Droplet/Aerosol transmission • societal behavioral change * All authors are listed in alphabetical order by surnames.

…”

Quantifying the COVID19 infection risk due to droplet/aerosol inhalation

“…A multi-physics solver known as CUBE [30,31] has been used for all the numerical simulations presented in this work. CUBE is a finite volume solver based on a hierarchical meshing framework known as the building cube method (BCM) [32].…”