Data-Layout Reorganization for an Efficient Intra-Node Assembly of a Spectral Finite-Element Method

Abstract

The Finite-Element Method (FEM) is routinely used to solve Partial Differential Equations (PDE) in various scientific domains. For seismic waves modeling, the Spectral Element Method (SEM), which is a specific formulation of the classical FEM approach, have gained significant attention for the last two decades. This is explained both from the very good numerical accuracy of this method and from the parallel performance of classical MPI-based implementations that scale up to several tens of thousands computing cores. Nevertheless, the trend for current processors with an increasing level of low-level parallelism requires significant efforts at the shared-memory level. One major bottleneck is coming from the standard FEM assembly phase that leads to significant amount of irregular memory accesses. This prevents any efficient automatic optimizations from the compiler for instance. In this paper, we extract a kernel from a spectral-element application dedicated to earthquake simulations in complex geological medium (EFISPEC code developed at BRGM, the French Geological Survey). We study the intranode behavior and we propose different levels of optimization (data-layout, manual vectorization, multithreading) to fully benefit from SIMD units and NUMA architectures. Experiments performed on Intel Broadwell architecture show that the proposed optimizations dramatically improve the intra-node performance of the mini-application. Moreover, our results show a good match with rooflines theoretical performance models. We believe that these optimizations are not specific to this mini-application and may be implemented in different SEM and FEM based solvers as well.

Performance Analysis of SIMD Vectorization of High-Order Finite-Element Kernels

“…The kernel extracted corresponds to the computation of the internal forces and represents a maximum of 90% of the total elapsed time. This paper extends previous works dealing with mono-core vectorization [11] and datalayout reorganization [12]. Our contributions in this paper are as follows:…”

“…The support of the AVX512 allows the hardware to compute 16 single precision floats by instruction. We recall from [12] that automatic optimizations provided by the compilers hardly reach 140 GFLOPS on both platforms, this represents less than 5% of the theoretical peak performance. Figure 9 compares the performance between 256bit and 512-bit SIMD instructions.…”

“…The counterpart is the limited efficiency as regards to cache memory. In [12] we have extended this approach to parallel FEM assembly on multi-core architectures by implementing colored packs of elements along with advanced vectorization strategies. Finally, one can change the data structure as detailed in [11].…”

“…The vectorization of our kernel is mainly described in [12], [11]. It is based on the use of intrinsics since we observed that compilers are not able to vectorize the code efficiently.…”