Seamless Integration of Parallelism and Memory Hierarchy

Abstract

We prove an analogue of Brent's lemma for BSP-like parallel machines featuring a hierarchical structure for both the interconnection and the memory. Specifically, for these machines we present a uniform scheme to simulate any computation designed for v processors on a v'-processor configuration with v' ≤ v and the same overall memory size. For a wide class of computations the simulation exhibits optimal O (v/v') slowdown. The simulation strategy aims at translating communication locality into temporal locality. As an important special case (v' = 1), our simulation can be employed to obtain efficient hierarchy-conscious sequential algorithms from efficient fine-grained ones

Translating submachine locality into locality of reference

“…The correctness of the resulting strategy was proved in [13]. Note that the simulation brings new clusters to the top of memory only when i s+1 < i s since, in order to simulate superstep s + 1 for an i s+1 -clusterĈ, superstep s must have been executed for all of the i s -clusters contained inĈ.…”

“…We adopt the same overall simulation strategy as the one presented in [13] for the HMM model, which we briefly recall below. The memory of the host machine is divided into blocks of µ cells each, with block 0 at the top of memory.…”

“…Instead, if i s+1 ≥ i s , no cluster swaps are performed, and the next round will simulate superstep s + 1 for the topmost i s+1 -cluster that is contained within the i s -cluster C currently residing on top of memory. Advancing the simulation unevenly for the different D-BSP submachines proved crucial in [13] to transform the submachine locality exhibited by P into temporal locality. However, in order to exploit also spatial locality effectively, as encouraged by the BT model, the actual implementation of the above scheme requires major modifications over the one in [13].…”

“…A more general study on the relation between parallelism and locality of reference has been initiated in [13], where we have shown how a more structured form of parallelism, such as the one exhibited by the Decomposable BSP (a clustered variant of BSP defined in [9]) can lead to the design of sequential algorithms that feature an optimal exploitation of temporal locality on arbitrarily deep memory hierarchies. More precisely, the main contribution of that work is a strategy that simulates a D-BSP algorithm on HMM with a slowdown which is merely proportional to the loss of parallelism, thus proving that no extra cost is incurred in accessing the much larger memory of the sequential machine.…”

“…More specifically, building on the results of [13] we devise an efficient strategy to simulate D-BSP algorithms on the BT model. The simulation reveals that efficiency in the BT model can be achieved starting from D-BSP algorithms exhibiting a much coarser level of submachine locality than the one needed by the simulation in [13], which required a decomposition into submachines strictly dependent on the access cost function. However, some examples provide evidence that a certain level of submachine locality must nonetheless be exhibited by the D-BSP algorithm in order to achieve optimal or quasi-optimal BT algorithms.…”