Code Generation and Optimization of Distributed-memory Dense Linear Algebra Kernels

Interfaces are key

“…The FLAME interfaces for indexing are also included to omit indexing in favor of reasoning about matrix partitions. The benefit of these interfaces is that parallelizing most sequential DLA algorithms in high-performance Elemental code is rote (this is described and automated in [6,7]). An expert needs to decide which distributions are efficient and how to redistribute between them.…”

“…In the case of DLA libraries, much of an experts' development work is rote thanks to good abstraction, and we can indeed automate it. In this section, we present the basics of DxT [6,7], which is used to encode expert knowledge about DLA interfaces. A system can then utilize that knowledge to generate high performance code.…”

“…With Elemental experts predict runtime to choose which parallelization schemes to use or optimizations to apply. Estimates are first-order approximations in terms of the amount of computation performed and the amount of data communicated between processes [6,7]. Thanks to the interfaces in Elemental, BLIS, and libflame, relatively rough cost estimates are good enough to guide experts without having to implement, compile, run, and time code.…”

Understanding performance stairs

“…We have automated the exploration of these spaces (by generating all implementations using a methodical process) and we evaluate the efficiency of each implementation via cost estimation. 1 This is how we find the best-performing algorithm that experts would intuitively select [17,18,19]. In all tests, generated code is the same or better than experts' hand-produced implementations.…”

“…We begin with a brief overview of how we generate the space of implementations for a given operation in the domain. Our approach is called Design by Transformation (DxT) -more details are given in [9,17,18,19,27]. …”