Studies on the energy and deep memory behaviour of a cache-oblivious, task-based hyperbolic PDE solver

Abstract

We study the performance behaviour of a seismic simulation using the ExaHyPE engine with a specific focus on memory characteristics and energy needs. ExaHyPE combines dynamically adaptive mesh refinement (AMR) with ADER-DG. It is parallelized using tasks, and it is cache efficient. AMR plus ADER-DG yields a task graph which is highly dynamic in nature and comprises both arithmetically expensive tasks and tasks which challenge the memory's latency. The expensive tasks and thus the whole code benefit from AVX vectorization, though we suffer from memory access bursts. A frequency reduction of the chip improves the code's energy-to-solution. Yet, it does not mitigate burst effects. The bursts' latency penalty becomes worse once we add Intel R Optane TM technology, increase the core count significantly, or make individual, computationally heavy tasks fall out of close caches. Thread overbooking to hide away these latency penalties becomes contra-productive with non-inclusive caches as it destroys the cache and vectorization character. In cases where memory-intense and computationally expensive tasks overlap, ExaHyPE's cache-oblivious implementation nevertheless can exploit deep, non-inclusive, heterogeneous memory effectively, as main memory misses arise infrequently and slow down only few cores. We thus propose that upcoming supercomputing simulation codes with dynamic, inhomogeneous task graphs are actively supported by thread runtimes in intermixing tasks of different compute character, and we propose that future hardware actively allows codes to downclock the cores running particular task types.

ExaHyPE: An engine for parallel dynamically adaptive simulations of wave problems

“…In general good scalability can be observed on up to 14 cores at higher orders. All ExaHyPE codes employ a hybrid parallelisation strategy with at least two MPI ranks per node [55]. Results for this hybrid parallelisation strategy are provided in Section 6.6.…”

Enclave Tasking for DG Methods on Dynamically Adaptive Meshes

“…If we study a linear variant of (1), we integrate the cell with the Cauchy--Kowalesvki procedure [14]. Here, the STP is significantly cheaper, though it still yields localized data access [8]. The time integration following the STP allows us to reuse the outcome data structure for all intermediate-in-time results.…”

“…Its computations per mesh cell are arithmetically intense, which is a property they share with many higher-order methods [25]. At the same time, DG's data access pattern however is very localized [11]---this helps to reduce the memory access stress [8,17,20,24,27]---and its exchange between cells along their connecting faces is conceptually simple. A combination of these two properties---high intensity to exploit vector units and dynamic adaptive mesh refinement (AMR) to invest where it pays off most---is a fit to predictions of what exascale software will have to look like [10].…”

TeaMPI—Replication-Based Resilience Without the (Performance) Pain

“…This allows us to single out a failing rank. The ∆t HB ensures that the system is not flooded with heartbeat messages and is not overly sensitive to small performance fluctuations [6].…”