Task-based Parallel Programming for Scalable Matrix Product Algorithms

Abstract

Task-based programming models have succeeded in gaining the interest of the high-performance mathematical software community because they relieve part of the burden of developing and implementing distributed-memory parallel algorithms in an efficient and portable way.In increasingly larger, more heterogeneous clusters of computers, these models appear as a way to maintain and enhance more complex algorithms. However, task-based programming models lack the flexibility and the features that are necessary to express in an elegant and compact way scalable algorithms that rely on advanced communication patterns. We show that the Sequential Task Flow paradigm can be extended to write compact yet efficient and scalable routines for linear algebra computations. Although, this work focuses on dense General Matrix Multiplication, the proposed features enable the implementation of more complex algorithms. We describe the implementation of these features and of the resulting GEMM operation. Finally, we present an experimental analysis on two homogeneous supercomputers showing that our approach is competitive up to 32,768 CPU cores with state-of-the-art libraries and may outperform them for some problem dimensions. Although our code can use GPUs straightforwardly, we do not deal with this case because it implies other issues which are out of the scope of this work.

On the Arithmetic Intensity of Distributed-Memory Dense Matrix Multiplication Involving a Symmetric Input Matrix (SYMM)

“…More specifically, we have chosen to rely on the sequential task flow (STF) model [3,4], where tasks are created sequentially and their mutual dependencies are automatically inferred through an analysis of their data access mode. This model is, for example, available in OpenMP (through the task directive and the depend clause), OmpSs [13] or StarPU [6], the runtime system we use in this work.…”

“…The effectiveness of this model on shared memory parallel computers has been proved by numerous works although it has been much more rarely considered in distributed-memory settings; early work on this topic is presented by YarKhan [28] and Agullo et al [3]. More recently, an extension of the STF model was proposed by Agullo et al [4] that allows for portable implementations of scalable algorithms for distributed-memory computers; we rely on this approach for the implementation of the proposed SYMM algorithms, as discussed in Section 3.…”

“…We therefore aim at designing a SYMM routine completely independent of the proposed mappings so that we can then effortlessly implement any non-trivial mapping. Task-based programming allows for such a separation of concerns [16,4]. In addition, we want to assess whether this is feasible with a code which is easy to write, read and maintain.…”

“…To this end, we decided, more specifically, to rely on the STF model [13,6] introduced in Section 2.3, which allows for writing a parallel code which resembles a sequential code one would naturally write. Finally, we do not want to trade off performance with elegance, and we thus rely on the latest developments for the scalability of the STF model [4] to design appropriate communication patterns while preserving the compactness of the expression. Without loss of generality, for the sake of conciseness, we restrict the presentation to the C ← AB + C case, where A is symmetric and only its lower part explicitly stored.…”

“…We also show that with the same amount of memory as GEMM, using an extension inspired by the 2.5D algorithms [1,25,24] (also referred to as 3D in the literature) allows SYMM to achieve a higher AI than GEMM. In order to assess whether the higher AI may translate into a higher performance, the resulting algorithms have all been implemented in the Chameleon dense linear algebra library [2] following the taskbased design proposed in [4]. This work only considers classical algorithms with a cubic computational complexity; Strassen-like variants [7] that can lead to reduced computation and communication volume are out of the scope of this paper.…”