Geographical Locality and Dynamic Data Migration for OpenMP Implementations of Adaptive PDE Solvers

ForestGOMP: An Efficient OpenMP Environment for NUMA Architectures

“…Another solution called next-touch is the generalization of the first-touch approach. It allows applications to ask the system to allocate or migrate a page near the thread that will perform the next access [19,23,26]. The next-touch policy thus can be used to redistribute data buffers to their new best locations between application steps.…”

“…Even if MARCEL and MAMI bring interesting knowledge of the current thread and data locations at runtime, large overheads are implied by the additional return to user-space (to run the signal handler before re-entering the kernel again for migration) and by the TLB flush on every processor during each mprotect (while another flush is already involved during page migration). On the other hand, SOLARIS has been offering an optimized kernel based next-touch implementation for a while and it is known to help applications significantly [19,23,26]. However, LINUX does not offer such a feature although it has spread to most high-performance computing sites nowadays.…”

“…To tackle irregular algorithms, [19,23,26] have studied the promising next-touch policy. Their approach however suffers from the lack of cooperation between the allocation library and the thread scheduler, and from not mastering the underlying memory architecture constraints.…”

Parallel Structured Adaptive Mesh Refinement

“…A better alternative is to have a coarse grained parallelization over the blocks, i.e., processing locally one or more blocks in each thread. OpenMP gives some support for load balancing through the schedule directive in loops (e.g., schedule dynamic) but as we also need to keep the data locality this approach is not sufficient for our goals [47]. A remedy is to mimic the MPI parallelization, i.e., to only use the parallel directive and check if the particular block belongs to this thread's partition before executing the code on the thread.…”

“…Markus Nordén et al made an experimental study under carefully controlled circumstances, to measure the effect of geographical locality on the performance of a parallel, block-wise SAMR code [47]. For the experiment, four nodes of a Sun Fire 15K system were used in dedicated mode.…”