Is 1.7 x 10^10 Unknowns the Largest Finite Element System that Can Be Solved Today?

Abstract

The hierarchical hybrid Grids (HHG) framework attempts to remove limitations on the size of problem that can be solved using a finite element discretization of a partial differential equation (PDE) by using a process of regular refinement, of an unstructured input grid, to generate a nested hierarchy of patch-wise structured grids that is suitable for use with geometric multigrid. The regularity of the resulting grids may be exploited in such a way that it is no longer necessary to explicitly assemble the global discretization matrix. In particular, given an appropriate input grid, the discretization matrix may be defined implicitly using stencils that are constant for each structured patch. This drastically reduces the amount of memory required for the discretization, thus allowing for a much larger problem to be solved. Here we present a brief description of the HHG framework, detailing the principles that led to solving a finite element system with 1.7 × 10 10 unknowns, on an SGI Altix supercomputer, using 1024 nodes, with an overall performance of 0.96 TFLOP/s, on a logically unstructured grid, using geometric multigrid as a solver.

Hierarchical hybrid grids: achieving TERAFLOP performance on large scale finite element simulations

“…Fortunately, as shown in Ref. [3], this does not adversely affect convergence, and this design allows for a much more efficient implementation of parallel communication, which is the topic of the next section.…”

“…The two-dimensional example is included because it is easier to visualize. 3: end for 4: for each edge do 5: copy from vertex interiors 6: apply operation to edge 7: copy to vertex halos 8: end for 9: for each face do 10: copy from vertex and edge interiors 11: apply operation to face 12: copy to vertex and edge halos 13: end for 14: for each element do 15: copy from vertex, edge, and face interiors 16: apply operation to element 17: copy to vertex, edge, and face halos 18: end for As a consequence of this approach, certain dependencies that exist between primitives of the same type are ignored. Fortunately, as shown in Ref.…”

“…As part of this analysis, we give a brief discussion of the performance of a code implemented using the hierarchical hybrid grids (HHG) library presented in Ref. [3], and show that good serial performance is necessary to be able to achieve reasonable scalability when performing large-scale simulations. We begin our investigation with an overview of the HHG framework.…”

Simulation of the human intracranial arterial tree

“…Use of high-order spectral/hp discretization is particularly valuable in simulations of transitional and turbulent flows Varghese et al 2007). A key numerical issue in three-dimensional large-scale simulations is the use of fast parallel solvers and preconditioners scalable to thousands of processors (Sherwin & Casarin 2001;Tufo & Fischer 2001;Bergen et al 2005;Grinberg et al in press).…”

FEAST—realization of hardware‐oriented numerics for HPC simulations with finite elements

“…Rüde and coworkers [6] pursue an approach similar to ours, using patchwise multigrid smoothers and hierarchical hybrid grids. A recent publication [7] reports good weak scalability for a 3D Poisson problem on a regularly refined unitcube domain for up to 9 170 cores (307 billion DOF).…”